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HIGHLIGHTS OF THE FINDINGS , 


The following findings highlight student and teacher outcomes of the study, as well as teacher 
feedback and program implementation. 

Student Outcomes: 

» Students participating in the TEC Summer Learning Program showed significant growth 
in their knowledge of mathematics vocabulary (41% gain from pre- to post-assessment), 
numeracy skills (20% gain from pre- to post-assessment), and phonics skills (17% gain 
from pre- to post-assessment). 

» Fourteen out of 16 participating teachers reported that students progressed or grew 
significantly in the area of “motivation and confidence to engage in learning activities” 
related to both mathematics and literacy. 

» All 16 teachers participating in the TEC Summer Learning Program reported that using 
the program increased student morale and enthusiasm in their summer school setting. 

Teacher Outcomes: 

» Fifteen out of 16 teachers reported that the TEC summer activities increased their moti¬ 
vation and confidence in leading groups. 

» Nine out of 16 teachers reported that they learned new instructional strategies for 

teaching mathematics and literacy. The three most common areas of learning were: new 
strategies for teaching vocabulary, incorporating technology into lessons, and combining 
different modes of learning to keep students engaged in a topic. 

Teacher Feedback: 

» Teachers reported that students found Session 7 of the DVD episodes to be the most 
appealing and engaging aspect of the TEC model, followed closely by Prankster Planet. 

» Teachers reported that they (teachers) found Session 7 of the DVD episodes to be the 
most appealing, interesting, and valuable aspect of the TEC model followed by the 
teacher-led activities and Jessica’s Word Wall. 

» Fifteen out of the 16 teachers reported that they would use TEC curriculum again, and 
15 out of the 16 teachers reported that they would recommend the curriculum to other 
summer school or afterschool teachers. 

Program Implementation: 

» Despite the diversity of the programs (with regard to class size, teacher experience, 
type of program, etc.), nearly all participating teachers implemented the TEC Summer 
Learning Program as planned. 

» Student outcomes were consistently positive, even with variations in the training 

teachers received and the delivery of the number of episodes of The Electric Company. 
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EXECUTIVE SUMMARY . 


This report describes a formative evaluation study of The Electric Company Summer Learning 
Program. The study is part of WestEd’s larger Ready to Learn formative evaluation activities. 
Funded by the U.S. Department of Education, the Ready to Learn grant supports the develop¬ 
ment of educational television and digital media targeted at preschool and early elementary 
school children and their families. Its general goal is to promote early learning and school 
readiness, with a particular interest in reaching children from low-income families. To create 
The Electric Company (TEC) Summer Learning Program, Sesame Workshop collaborated with 
Ready to Learn joint grantees, the Corporation for Public Broadcasting (CPB) and the Public 
Broadcasting Service (PBS), in their effort to deliver a next-generation educational ecosystem of 
integrated transmedia content in literacy and numeracy that is aligned with rigorous academic 
frameworks and research. Transmedia storytelling is the technique of telling stories across 
multiple platforms and formats using current digital technologies. The TEC Summer Learning 
Program is a transmedia curriculum that presents a narrative story to students that is presented 
across multiple forms of media, including television episodes and an online gaming experience. 
The evaluation of the TEC program is important because it contributes to our knowledge of the 
usefulness of transmedia in educational settings. In addition, it provides valuable feedback to 
producers, researchers and funders as they plan and create new transmedia products. 

The current study focuses on the implementation and educational effectiveness of The Electric 
Company (TEC) Summer Learning Program. TEC’s producer, Sesame Workshop, created the 
TEC Out of School Time (OST) curricular model. TEC provides 36 hours of content based on 
mathematics vocabulary, numeracy and literacy skills. This formative study provides feedback to 
CPB, PBS, and Sesame Workshop on successes and areas of improvement for the program, and 
examines student outcomes of the summer program. The study took place from March 2011 to 
September 2011 and includes student participants, ages 6 to 8, from summer programs across 
the country. 

The TEC Summer Learning Program curriculum offers teachers explicit directions and scripts 
detailing the steps to implement multi-media activities geared towards children ages 6 to 8 in 
summer learning environments that extend the curriculum through games, view-and-do activi¬ 
ties, online adventures and video clips. The TEC curriculum provides corresponding activity 
sheets to help support DVD episodes of Season Three of the all-new The Electric Company 
television show and the accompanying Prankster Planet online experience. Prankster Planet is a 
transmedia gaming experience that drills more deeply into math-related vocabulary and math 
skills through multi-layered gameplay and story. 
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During the study, content was implemented in 5- to 6-week summer programs for 90 minutes 
a day, for a total of 24 days. 1 This 36-hour curricular model includes: scripted group facilitation, 
DVDs of 12 new TEC episodes (viewing two episodes per week), targeted broadband activities, 
and individual or peer activities. The logic model that guides the study posits that significant 
fidelity to the intended implementation of the TEC summer curriculum will increase 6- to 
8-year-old student skills, motivation to learn, and knowledge and attitudes around literacy and 
numeracy, particularly with respect to targeted mathematics vocabulary. 

The study utilized a mixed-method design to examine how the use of the TEC Summer Learning 
Program affects students’ skills, motivation, knowledge, and attitudes toward literacy and 
numeracy. The study included data from teachers and students, including student usage data 
for Prankster Planet that is collected through Google Analytics. 


RESEARCH METHOD 

The intervention included the use of the TEC Summer Learning Program by 16 teachers in their 
summer school classrooms during the summer of 2011. A mixed-method study design was used 
to address the following research questions. 

» Are teachers implementing the TEC Summer Learning Program as planned? Are there 
any obstacles to implementation? What variables contribute to fidelity of implementa¬ 
tion? (e.g., number of children, children’s initial level of knowledge, years of teacher’s 
experience, type of program). 

» Do students who participate in the TEC Summer Learning Program increase skills in 
select mathematical domains? 

» Do students who participate in the TEC Summer Learning Program acquire the math¬ 
ematics vocabulary targeted by the program? 

» Do students who participate in the TEC Summer Learning Program increase their com¬ 
prehension of connected text? Do they learn strategies that good readers use to under¬ 
stand connected text? 

» Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to literacy? 

» Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to mathematics? 

» Do teachers who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence leading groups? 

» Do teachers who participate in the TEC Summer Learning Program report an increase in 
morale and enthusiasm in their summer school setting? 


1 Several shorter programs omitted two episodes chosen by Sesame Workshop for a 30-hour, 
90 minutes a day curricular model for a total of 20 days. 
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» Do teachers who participate in the TEC Summer Learning Program learn new instruc¬ 
tional strategies in teaching mathematics and literacy? 

» Which aspects of the TEC model do students find the most (and least) appealing, inter¬ 
esting, and educational? 

» Which aspects of the TEC model do teachers find the most (and least) appealing, inter¬ 
esting, and valuable, particularly in regard to learning content and increasing motivation? 

» Which aspects, if any, of the TEC model (including professional development) do the 
teachers think might need to be refined, improved, or modified? Would teachers use the 
summer curriculum again? Would they use all or part of the curriculum in their regular 
classroom? Why or why not? 

The following data were collected over the course of the study. 

» Standards/Skills Aligned Survey. The student survey included measures of student 
achievement including researcher-developed items and released math items from the 
California Standards Test. (Items content aligned with the TEC curriculum.) In addition, 
this survey included items measuring academic motivation and confidence. This survey 
was given to students at the start of their summer program, and again at the end. 

» Student Interview. A series of student interviews took place at each site. Students were 
asked about their experiences with the TEC Summer Learning Program and about the 
content presented in the curriculum as well as the meaning of targeted mathematics 
vocabulary. 

» Teacher Intake Survey. The teacher intake survey addressed teacher demographics, 
teacher experience, and school and classroom variables. 

» Teacher Exit Survey. The teacher exit survey addressed teacher perceptions of student 
learning, motivation, and confidence. Teachers provided feedback on the TEC Summer 
Learning Program in light of related research questions. 

» Student Usage Data From Google Analytics. WestEd imported the individual student 
click data and Flash tags from the students who used Prankster Planet. These data were 
analyzed to measure students’ time on task in various aspects of the game, and their 
success in embedded assessments in the game. 

» Classroom Observation. Researchers visited study sites several times during the course 
of the summer. Researchers gathered data about fidelity of implementation at sites, and 
collected data related to the research questions. 

» Teacher Interview. Researchers conducted a mid-visit and final interview with each par¬ 
ticipating TEC teacher. The interview questions were developed to address the research 
questions, check for fidelity of implementation, and gather feedback on the curriculum. 
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STUDY PARTICIPANTS 

Participants included students ages 6 to 8 years, specifically students between the first and 
second grades 2 , enrolled in 12 diverse 5- to 6-week summer programs throughout the United 
States for a total of 152 student participants. Participants also included 16 teachers at the 
12 summer programs. Teachers implemented the TEC Summer Learning Program model in their 
classrooms. Participating summer programs enrolled a high proportion of children from low- 
income families and English language learners. Programs were school-site based and located 
in rural, urban, and suburban areas in California, New Mexico, Utah, New Jersey, Ohio and 
Washington, D.C. 

All students in the study participated in their summer school program as planned. Students were 
provided with TEC content for 90 minutes a day, four or five days a week, depending on the 
length of their summer program. Teachers received stipends totaling $1000. Each participating 
site received $500. 


ANALYSIS 

Data were analyzed to address the study’s research questions. The qualitative data (interview 
data and open-ended items on surveys) were coded and analyzed using the ATLAS.ti and 
HyperResearch software programs, which allow researchers to assign codes to qualitative data 
to identify trends and explore relationships across data elements. The initial coding scheme was 
based upon the model of program implementation identified by TEC and specific aspects of the 
research questions. 

Descriptive statistics were also used to illustrate changes in teachers and students as a result of 
the program. We used inferential analyses, such as paired samples t-tests, to determine whether 
the students and teachers showed significant growth between the pre-test and post-test sur¬ 
veys. Furthermore, correlations were used to determine if significant associations exist between 
usage and growth on the surveys. 


FINDINGS 

WestEd employed quantitative and qualitative data analyses to address the study's research 
questions. In general, the TEC Summer Learning Program was well received in study summer 
school programs. Teachers highly valued the program and all student outcomes were positive. 
Specific findings by research question include: 

» Are teachers implementing the TEC Summer Learning Program as planned? Are 
there any obstacles to implementation? What variables contribute to fidelity of 


2 One site included students between the second and third grades. 
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implementation? (e.g., number of children, children’s initial level of knowledge, years of 
teacher’s experience, type of program). 

— Despite the diversity of the programs (with regard to class size, teacher experi¬ 
ence, type of program, etc.), nearly all participating teachers implemented the 
TEC Summer Learning Program as planned. 

— Student outcomes were consistently positive, even with variations in the training 
teachers received and the delivery of the number of episodes of The Electric 
Company. 

— The obstacles to implementation reported by teachers included: inconsistent 
student attendance in summer programs; technical difficulties at sites, especially 
with regard to running Prankster Planet on school computers; student lack of 
computer knowledge; summer program structure (e.g., class size either too small 
or too big, computer access too far from classroom, etc.); and teacher issues with 
either the length of time allotted for activities in the curriculum or the content of 
the curriculum being too challenging for some students. 

» Do students who participate in the TEC Summer Learning Program increase skills in 
select mathematical domains? 

— Students showed a 20% gain in their numeracy skills from pre- to 
post-assessment. 

» Do students who participate in the TEC Summer Learning Program acquire the math¬ 
ematics vocabulary targeted by the program? 

— Students showed a 41% gain in their mathematics vocabulary from pre- to 
post-assesment. 

» Do students who participate in the TEC Summer Learning Program increase their com¬ 
prehension of connected text? Do they learn strategies that good readers use to under¬ 
stand connected text? 

— Students showed a 17% gain in their phonics skills from pre- to post-assessment. 

— Twelve out of 16 teachers reported that students had progressed in the area of 
“learned and used strategies to understand connected text.” 

» Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to literacy? 

— Fourteen out of 16 teachers reported that the students progressed or grew 
significantly in the area of “motivation and confidence to engage in learning 
activities related to literacy.” 

» Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to mathematics? 

— Fourteen out of 16 teachers reported that the students progressed or grew 
significantly in the area of “motivation and confidence to engage in learning 
activities related to mathematics.” 

» Do teachers who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence leading groups? 
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— Fifteen out of 16 teachers reported that the TEC summer activities increased their 
motivation and confidence in leading groups. 

» Do teachers who participate in the TEC Summer Learning Program report an increase in 
morale and enthusiasm in their summer school setting? 

— All 16 teachers participating in the TEC Summer Learning Program reported that 
using the program increased student morale and enthusiasm in their summer 
school setting. 

» Do teachers who participate in the TEC Summer Learning Program learn new instruc¬ 
tional strategies in teaching mathematics and literacy? 

— Nine out of 16 teachers reported that they learned new instructional strategies 
for teaching mathematics and literacy. The three most common areas of learning 
were: new strategies for teaching vocabulary, incorporating technology into 
lessons, and combining different modes of learning to keep students engaged in 
a topic. 

» Which aspects of the TEC model do students find the most (and least) appealing, inter¬ 
esting, and educational? 

— Teachers reported that students found Session 1 of the DVD episodes to be 
the most appealing and engaging aspect of the TEC model, followed closely 
by Prankster Planet, and the activity worksheets to be the least engaging and 
appealing aspect. 

» Which aspects of the TEC model do teachers find the most (and least) appealing, inter¬ 
esting, and valuable, particularly in regard to learning content and increasing motivation? 

— Teachers reported Session 7 of the DVD episodes to be the most appealing, 
interesting, and valuable aspect of the TEC model followed by the teacher-led 
activities and Jessica’s Word Wall. They found the activity worksheets were the 
least appealing, interesting, and valuable. 

» Which aspects, if any, of the TEC model (including professional development) do the 
teachers think might need to be refined, improved, or modified? Would teachers use the 
summer curriculum again? Would they use all or part of the curriculum in their regular 
classroom? Why or why not? 

— Teachers suggested that the professional development training could be 
improved in a variety of ways, including: allow teachers to see the curriculum 
being implemented in an actual classroom, dedicate the webinars to questions 
and answers only (i.e. have teachers review the materials ahead of time), and 
schedule webinars later in the day for teachers on the west coast. 

— The aspect of the TEC model that evoked the most comments with regard to 
what might be improved was the worksheets. Comments and suggestions for 
improvement included: worksheets should include phonics or literacy activities 
and worksheets were either too far below or above the students’ skill levels. 

— Fifteen out of the 16 teachers reported that they would use TEC curriculum again, 
and 15 out of the 16 teachers reported that they would recommend the cur¬ 
riculum to other summer school or afterschool teachers. 


c 


WestEd // Evaluation of The Electric Company Summer Learning Program // EXECUTIVE SUMMARY 


page 


7 





— Thirteen out of 16 teachers reported that they would use all or part of the cur¬ 
riculum as part of their regular classroom activities. All three of the teachers who 
said they would not use it as part of regular classroom teaching stated that, while 
they would not use it as their primary teaching tool, they would use portions of 
the TEC curriculum to reinforce concepts already taught in the classroom. 


CONCLUSION 

The results suggest that the TEC Summer Learning Program can be a valuable tool for educa¬ 
tors to boost student interest in learning and to promote academic achievement in specific 
content areas related to numeracy and literacy. The evaluation focused on the use of the TEC 
Summer Learning Program in summer school programs serving students ages 6 to 8 years from 
low-income families. Students and teachers in the study found TEC to be engaging throughout 
the entire 24 days included in the model. The findings from this mixed-methods formative study 
suggest that the TEC Summer Learning Program can increase children’s interest in learning 
in specific subjects, and can contribute to increased academic achievement in numeracy and 
literacy. In addition to gains in student outcomes, teachers who used the curriculum reported 
gains in their own learning, particularly around ways to deliver instruction and technology use. 

This comprehensive mixed methods study produced important findings about the TEC Summer 
Learning Program and its effects on students, particularly in relation to its effects on students' 
classroom engagement, interest in academic content, and student learning in the summer school 
environment. The TEC curriculum is unique in that it is combines different modes of instruc¬ 
tion (television show episodes, small and large group hands-on activities, and online gaming) 
to engage students in the academic content. It is intended to be delivered in the Out of School 
Time (OST) environment, yet support students’ learning of core academic skills. This study is 
important in that is shows that an innovative transmedia summer program can fit seamlessly 
into the summer school environment and support schools as they strive to engage and teach 
students from widely diverse backgrounds and ability levels. 

The current study highlights the promise of transmedia in OST environments for student learning 
and success. The study also shows that teachers can successfully use and embrace a transmedia 
curriculum. The findings suggest that further study is warranted to explore how students learn in 
the TEC curricular environment. Topics for future study include: the added value of each aspect 
of the TEC Summer Learning Program (television show episodes, small and large group hands- 
on activities, and online gaming), how teachers expand on the TEC curriculum, how teachers use 
aspects of the TEC curriculum in their regular classrooms, expansion of the TEC curriculum to 
include new digital devices, and use of the TEC curriculum in other OST environments, such as in 
after-school programs. 
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The Electric Company, a popular educational television show, was introduced over 30 years ago 
and centered around the goal of showing children that reading can be fun. In 2009, the all- 
new The Electric Company, produced by Sesame Workshop, returned to public television with 
a new cast of characters, cartoons, and songs. According to its producer, Sesame Workshop, 
the show aims to entertain children between the ages of 6 and 9 while simultaneously teaching 
crucial areas of literacy that are challenging for struggling readers. Season Three of the new The 
Electric Company expands to the areas of numeracy and the vocabulary of mathematics. 

This formative evaluation study of The Electric Company (TEC) Summer Learning Program 
is part of WestEd’s larger Ready to Learn formative evaluation activities. Funded by the U.S. 
Department of Education, the Ready to Learn grant supports the development of educational 
television and digital media targeted at preschool and early elementary school children and 
their families. Its general goal is to promote early learning and school readiness, with a particular 
interest in reaching children from low-income families. 

The Electric Company (TEC) Summer Learning Program involves a transmedia experience that 
focuses on the mathematics vocabulary, and mathematics and literacy content embedded in 
Season Three of the new The Electric Company. Prankster Planet, a new online gaming experi¬ 
ence, was designed to support the Season Three theme of mathematics vocabulary, and is an 
integral part of the TEC Summer Learning Program. 

This report describes a formative evaluation study addressing the effects of students’ and 
teachers’ use of the TEC Summer Learning Program on students’ mathematics vocabulary, and 
numeracy and literacy skills. The report also provides feedback to Sesame Workshop and Ready 
to Learn partners, including CPB and PBS, on successes and areas for program improvement. As 
a part of the evaluation, the following research questions were addressed: 

1. Are teachers implementing the TEC Summer Learning Program as planned? Are there 
any obstacles to implementation? What variables contribute to fidelity of implementa¬ 
tion? (e.g., number of children, children’s initial level of knowledge, years of teacher’s 
experience, type of program). 

2. Do students who participate in the TEC Summer Learning Program increase skills in 
select mathematical domains? 

3. Do students who participate in the TEC Summer Learning Program acquire the math¬ 
ematics vocabulary targeted by the program? 
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4. Do students who participate in the TEC Summer Learning Program increase their com¬ 
prehension of connected text? Do they learn strategies that good readers use to under¬ 
stand connected text? 

5. Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to literacy? 

6. Do students who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence to engage in learning activities related to mathematics? 

7. Do teachers who participate in the TEC Summer Learning Program increase their motiva¬ 
tion and confidence leading groups? 

8. Do teachers who participate in the TEC Summer Learning Program report an increase in 
morale and enthusiasm in their summer school setting? 

9. Do teachers who participate in the TEC Summer Learning Program learn new instruc¬ 
tional strategies in teaching mathematics and literacy? 

10. Which aspects of the TEC model do students find the most (and least) appealing, inter¬ 
esting, and educational? 

11. Which aspects of the TEC model do teachers find the most (and least) appealing, inter¬ 
esting, and valuable, particularly in regard to learning content and increasing motivation? 

12. Which aspects, if any, of the TEC model (including professional development) do the 
teachers think might need to be refined, improved, or modified? Would teachers use the 
summer curriculum again? Would they use all or part of the curriculum in their regular 
classroom? Why or why not? 

The study addressed the research questions with pre-test and post-test surveys of student 
participants, teacher and student interview data, website tracking data, classroom observa¬ 
tions, and teacher intake and exit surveys. The study utilized a design in which summer school 
programs serving 6- to 8 year-old children between the first and second grades used the TEC 
Summer Learning Program curriculum for 30-36 hours over the course of their summer school 
program. Summer programs were recruited from across the United States. 


OVERVIEW OF THE STUDY 

The evaluation of the TEC Summer Learning Program took place from March 2011 to 
September 2011 and addressed the effectiveness of the program in increasing students’ math¬ 
ematics vocabulary, and numeracy and literacy skills. Another purpose of the evaluation was to 
provide feedback to Sesame Workshop, the producer of the TEC Summer Learning Program, 
and Ready To Learn partners, including CPB and PBS, in order to identify successes and areas of 
improvement for the program. 

The intervention consisted of students in diverse summer programs across the United States 
receiving 36 hours of content including: DVDs of the 12 episodes of Season Three of the all- 
new The Electric Company, small and large group activities, games, and access to Prankster 
Planet—a transmedia gaming experience that supports the Season Three theme of mathematics 
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vocabulary. Teachers received two hours of online professional training on how to implement the 
TEC Summer Learning Program curriculum as well as a detailed curriculum guide. 

Summer programs were identified for the study based on the length of their program, their 
access to computers, teachers’ knowledge of and comfort with using technology in the class¬ 
room, and the diversity of location. Sites were situated in urban, suburban and rural areas. Most 
summer programs’ enrollment included a high percentage of students from low-income families, 
including English language learners, drawn from diverse populations. 

Data were collected from participants at three points in time. Students were given a written 
standards/skills aligned survey and interviewed during the first few days of their program. The 
written survey and interview addressed both mathematics content and vocabulary targeted 
by the TEC curriculum as well as students’ attitudes about reading and mathematics. Site visits 
were made to most sites during the middle of the summer school programs both to observe and 
document fidelity of implementation and student engagement. Researchers observed the speci¬ 
fied 90-minute curriculum and interviewed the teacher. Teacher interviews addressed modifica¬ 
tions to the curriculum and general issues surrounding curriculum implementation. During the 
intervention, students’ use of the Prankster Planet site was tracked in Google Analytics. At the 
end of the summer school program, students were given a post-survey and were interviewed (as 
they were at the beginning of each program). The students’ post interviews included questions 
about specific elements of the TEC Summer Learning Program. Also at this time, teacher post 
interviews were conducted. All teachers were administered an exit survey once they had com¬ 
pleted their summer program. Exhibit 1 shows the types of data collected from participants over 
the course of the study. 
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Exhibit 1: Overview of Data Collection 


r 




PRE-INTERVENTION DATA COLLECTION 

Student Standards/Skills Aligned Pre-Test Surveys and Interviews 
Teacher Intake Surveys 



INTERVENTION 

Prankster Planet Student Usage Data 
Site Visits, Researcher Observations 
Teacher Interviews 



POST-INTERVENTION DATA COLLECTION 

Student Standards/Skills Aligned Post-Test Surveys and Interviews 
Prankster Planet Student Usage Data 
Teacher Interviews 
Teacher Exit Surveys 

v_I_ ) 

METHODOLOGY 

In this section, we describe recruitment, study requirements, study participation in the summer 
program, measures, and data analysis. 


RECRUITMENT 

The recruitment process for the TEC Summer Learning Program study began on April 19, 2011 
and concluded on June 27, 2011, when the last summer school program was admitted to the 
study. Upon obtaining IRB approval for the study, WestEd researchers immediately began 
recruitment of six-week summer programs. WestEd researchers invited more than approximately 
2000 district leaders, principals, and teachers across the United States to either complete 
an online application for the TEC Summer Learning Program study or to pass the informa¬ 
tion along to the appropriate summer program contact. The district leaders, principals and 
teachers were chosen from WestEd’s extensive database of state, district, and school contacts 
and were targeted for their work in low-income, high-need schools and districts. Initially, only 
six-week or longer summer programs were targeted, as that was the desired length identified by 
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Sesame Workshop to implement the 36-hour, 90-minutes per day, 4 days per week curriculum. 

It became clear early on in the recruitment process that 5-week programs would have to be 
included, as a limited number of six-week long programs with the proper criteria were able to 
enroll. Another issue regarding finding appropriate six-week long summer programs was that 
larger districts, which tended to have the longer summer programs, required extensive research 
applications that required several months for approval. Sesame Workshop gave permission to 
drop two of the twelve episodes from the curriculum for a reduced 30-hour, 90-minutes per day 
over 20 days model. 

Over 200 summer program directors, principals, and teachers responded to WestEd’s invitations 
to participate in the study. Potential summer school sites were screened for length of program, 
technological capability, access to computers, and technological experience of teachers. Both 
potential summer program directors/principals as well as potential teachers were interviewed by 
phone in order to ascertain whether the site and teacher would be a good match for the cur¬ 
riculum model. 

Summer program sites eligible to participate in the study were programs that served classes of 
“rising” second graders (students between the first and second grades of school), had at least 
one Internet-ready computer for every two students enrolled in the study, and were affiliated 
with a local school. Eligible summer school teachers were sent a teacher consent form to sign 
and return to WestEd, a set of student consent forms (in English and Spanish, where requested) 
to be distributed to their students, and The Electric Company Summer Learning Program cur¬ 
riculum, along with accompanying DVDs. Principals/directors of the eligible summer programs 
were also sent a Memorandum of Understanding (MOU) to sign and return to WestEd. 

By June 27, 2011, WestEd had received MOUs from principals/directors of 12 diverse summer 
programs and 16 consent forms from teachers in those programs. 

The ideal class size identified for participating sites was 15 students to 1 adult. The class sizes of 
participating teachers varied widely from 5 to 1 to 25 to 1. The mean average ratio of students to 
participating teacher was 9 to 1. Over the course of the study, WestEd received consent forms 
from 152 students. 

STUDY REQUIREMENTS 

The requirements of the study were clearly outlined in a Memorandum of Understanding with 
each participating summer school program, and in student and teacher consent forms. These 
forms stated the expectations of all participants. 

All sites agreed to distribute and collect study consent forms for students in their summer 
school program. In addition, all sites agreed to administer the student pre-test surveys and post¬ 
test surveys. Teachers agreed to attend live or watch archived professional development webi¬ 
nars and to implement the TEC Summer Learning Program curriculum as intended. Sites also 
agreed to register students on Prankster Planet through the WestEd portal page and to note 
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and send to WestEd students’ usernames and passwords so that student usage could be tracked 
through Google Analytics. All sites were expected to participate in site visits, student and 
teacher interviews, and to respond to email and telephone calls related to study implementation. 


STUDY PARTICIPATION IN SUMMER SCHOOL SITES 

In this section, we describe the intervention at summer program sites, including a description 
of the TEC Summer Learning Program curriculum, the training of the teachers, and profiles of a 
sample of sites. 


The TEC Summer Learning Program Curriculum 

This study addressed summer program students’, ages 6 to 8 years, exposure to the TEC 
Summer Learning Program curriculum, which included: viewing DVDs of 12 episodes of Season 
Three of the all-new The Electric Company, scripted facilitation of small and large group activi¬ 
ties surrounding the content in the episodes, and time spent on Prankster Planet, a transmedia 
gaming experience that supports the Season Three episodes. 

Children access Prankster Planet online through the pbskids.org website and complete “mis¬ 
sions” using what they learn from The Electric Company episodes. The gaming experience 
includes 12 two-minute animated segments, 12 multi-level online quests, 60 mini-games, an 
avatar creator, and a rewards system to encourage repeat play. The experience involves children 
creating an avatar of themselves and then working online to “help” animated versions of charac¬ 
ters from The Electric Company stop the Pranksters from stealing all the words from the Earth. 
They do this by completing 12 missions centered on the vocabulary from each of the 12 episodes. 
The words are woven into the online experience through character dialogue, vocabulary chal¬ 
lenges, and mini games that provide children with hands-on applications of the math concepts 
introduced in the episodes. 

The TEC curriculum is divided into 24 sessions (two sessions per episode). Each session consists 
of a simple, repetitive structure designed to reinforce student learning from week to week. 

Session 7 consists of five different activity blocks: Watch and Learn; Stretch and Assess; Get 
Electric; Show Off Your Skills; and Feel the Power. During Watch and Learn, students are intro¬ 
duced to the session’s Electric Words and key mathematics concepts while viewing a 30-minute 
The Electric Company episode as a large group. Stretch and Assess is a 5-minute break where 
students can get up and move around while reviewing the target words for the session. To Get 
Electric, students interact with the target words in a hands-on game. The Show Off Your Skills 
segment has students dividing up into small groups to play Prankster Planet, show their vocabu¬ 
lary knowledge at Jessica’s Word Wall, and complete Workout sheets. Students wrap up Session 7 
with a Feel the Power review of the Electric Words and a handout of games to play at home. 

The Show Off Your Skills segment of each session represents a key component of the curriculum. 
Jessica’s Word Wall is not only a display of the target words for each session, but an activity 
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center where students can challenge each other and themselves to learn vocabulary through 
dance, drawing, and storytelling quizzes. The Workout sheets extend this learning by reminding 
students of the Electric Words and their definitions, and asking them to use the words in com¬ 
plete sentences. Playing Prankster Planet reinforces the mathematics skills and vocabulary in 
the curriculum, as students solve puzzles and work together to keep the world from becoming 
wordless. 

Session 2 follows a similar activity block format. Students begin the session with a review, going 
over the Electric Words as a group, incorporating a call-and-response chant. During Pause and 
Play, students watch the same The Electric Company episode as in Session 1, but the episode is 
paused at key moments to allow for group questions, games, and activities to reinforce learning 
Students again participate in Show Off Your Skills, rotating through the small groups. Session 2 
ends with Feel the Power, reviewing the Electric Words from current and past sessions. 


Teacher Training 

Prior to the start of their summer programs, teachers were offered four possible times (two 
per part) to attend Parts 1 and 2 of a live orientation webinar to prepare them for implementing 
the TEC Summer Learning Program curriculum as intended. Those teachers who were unable 
to attend one or both parts of the live webinar were sent a link to an archived webinar (one for 
each part) to view at their convenience. Each teacher was also sent DVDs of Sessions 1 and 2 
of all 12 episodes of Season Three of the new The Electric Company and a detailed curriculum 
guide containing an overview of the program, detailed lesson plans including scripted facilitation 
of various activities, and masters of the "Workout Pages”—companion activity sheets. Sesame 
Workshop developed and hosted the orientation webinars, and a member of the WestEd team 
was present at each of the sessions to answer any questions pertaining to the research side 
of the study. Part 1 of the orientation was an overview of the The Electric Company project, 
and informed participants of the educational philosophy and goals behind the development of 
both the show and the curriculum. Part 2 used a particular episode of the show to drill down 
into implementation of specific aspects of the curriculum. Through the lens of one episode, 

Part 2 emphasized how to best facilitate all the different elements of the curriculum. All but 
two teachers were able to attend the live webinars or watch some combination of the live and 
archived webinars. 

Profiles of Sites 

The following profiles highlight the diversity of sites enrolled in the TEC Summer Learning 
Program study. Sites differed on many factors, including: geography, class size, student demo¬ 
graphics, teacher experience, structure of the summer program and diversity of the site. Though 
implementation of the TEC curriculum may have differed due to some of these factors, partici¬ 
pants at all sites clearly benefited from the experience. 
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Site A 

This summer program, set in a suburban East Coast city, provided a 6-week, 5 days per week, 

6 hour per day summer enrichment program for a diverse, low-income, urban, at-risk student 
population. During the regular school year, the site is an elite, private, K-8 school, but in the 
summer, the facilities house a program for public school students. The program identifies the 
following goals: to prevent or reduce summer learning loss, to foster lifelong desire for learning, 
to provide cultural experiences and social growth, and to ensure that the students learn to swim. 
Each morning the students took an hour-long bus ride to the school, where they immediately 
were served a nutritious breakfast. The TEC Summer Learning Program was used in the morn¬ 
ings, reserved for “creative academics,” while the afternoons were reserved for art, swimming, 
athletics, and computer education. Fridays were reserved for cultural trips. This program was 
an excellent fit for the TEC Summer Learning Program study, as it already had the embedded 
academic time slot that could easily be adapted for the TEC curriculum. The study class had 
15 students between their first and second grade years. There was one veteran teacher for the 
class, who had several teenaged “helpers” in the classroom on any given day. The class had 8 
computers in the room. The teacher for this site did not attend either live session of the orienta¬ 
tion webinars, nor did she watch the archived webinars. She stated in her exit survey: "I thought 
reviewing the DVDs before I presented them was enough for me to be prepared for each lesson.” 
Though she did make some modifications to the curriculum (i.e. changed small group activities 
to whole group activities for “management” purposes), she was extremely positive about the 
curriculum and impressed with the level of engagement of her students throughout the pro¬ 
gram. She wrote that she intends to use the program again next summer. 

Site B 

This summer program, set in a rural county of a western state, provided a 5-week, all day enrich¬ 
ment program for elementary students from its large district, which includes 16 elementary 
schools. The students were diverse—only 50% Caucasian, and included a small number of 
English language learners. A district administrator stated that about half of the students quali¬ 
fied for free and reduced price lunch during the school year. The morning program combined 
elementary students of all ages for indoor and outdoor activities. After lunch, the teacher pulled 
out the “rising” second grade students in order to facilitate the TEC Summer Learning Program 
curriculum. The students used the school library as their classroom, as that is where approxi¬ 
mately ten school computers were housed. The teacher reported that attendance for the overall 
program had been irregular. The study class had anywhere from 5 to 10 students on any given 
day, all of whom were very much engaged by the curriculum as reported by their teacher. Once 
the study class completed their 90-minute TEC curriculum for the day, they re-joined the rest of 
the mixed grade levels summer program. 

The teacher at this site had only been teaching for two years, but was very excited about the 
program. He said, “There were kids that really grew attached to this part of our summer school 
program.” He also reported that he learned new instructional strategies in teaching mathematics 
and literacy and “enjoyed many of the songs and short skits that taught concepts.” 
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Site C 

This summer program was set on an American Indian reservation in a southwestern state. The 
program provided a 5-week, all day free enrichment program for elementary students on the 
reservation. All the participating students were of American Indian descent, and the school 
placed an emphasis on promoting the American Indian language and culture. The participating 
teacher stated that all the students qualified for free and reduced price lunch during the school 
year. Two adult classroom aides assisted the teacher with activities. Attendance for this summer 
program dropped drastically by the end of the 5-week program. At the beginning of the 
summer, there were 22 students enrolled in the program and by the end there were only 8 stu¬ 
dents. The school made efforts to follow up with students who dropped out of the program, but 
the attempts were unsuccessful. The program began at 8:30 am with the Voyager literacy cur¬ 
riculum and self-directed reading. Later in the morning, the teacher began the 90-minute lesson 
with The Electric Company. The first half of the curriculum was delivered before lunch, which 
included watching the episode and whole class activities. After lunch, the students would break 
out into small groups and play Prankster Planet or work on the worksheet. The students worked 
individually on the four functioning computers in the classroom. During the summer program, 
the school administered a school-mandated pre- and post-student assessment. The teacher 
reported an increase in the literacy scores on a school literacy assessment and attributed part 
of the student growth to the TEC curriculum. Overall, the teacher and students had positive 
feedback about the TEC curriculum. The teacher commented on viewing the episodes: “[The 
students] would sometimes ask me if they could watch the next episode on the same day[...] I 
get a kick out of them because on the second part they would yell, “Pause,” when they would 
see the little two lines on the top. I’m there with them, but they’re still screaming it because they 
loved [the episodes].” 

Site D 

This summer program was set in a rural county of a midwestern state and provided a 5-week, 
full-day enrichment program for elementary students. The program was designed to review 
academic content from the previous year as well as introduce new concepts to be addressed in 
the upcoming school year based on the state academic content standards. The students at the 
school site were very diverse, with less than 10% being Caucasian. The majority of these students 
were the children of migrant farm workers, and approximately 25% were considered English 
language learners. The study class had six students and was housed in a first grade classroom. 
Students traveled to a computer lab and also had access to a small library area. 

The teacher at this site noted that students “loved connecting actions with the vocabulary 
words”. These students enjoyed the episodes so much that they voted to watch the two epi¬ 
sodes that they had been scheduled to skip ( Wiki Wiki Walter and Prankster Holiday) for fun on 
the last day of their program. The teacher also noted that she is looking forward to using the 
curriculum again. 

Site E 

This summer program was set in an urban, low-income school district in a western state. The 
charter school provided a morning enrichment program for elementary students. The program 
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targeted a diverse group of at-risk students, which were of African American, Hmong and 
Hispanic descent. A majority of the students were classified as English language learners. There 
was only one female student in the class of 14 students. Each day the school offered free break¬ 
fast and lunch for the students. Since the duration of the school day was relatively short, the 
majority of the instructional time was dedicated solely to the TEC curriculum. During the class¬ 
room observation, the researcher noted how closely aligned the teacher's implementation of 
the curriculum was to the TEC lesson plans. The teacher often used the exact language scripted 
in the curriculum guide with his students. The classroom had two functioning computers for 
students to use for Prankster Planet. Due to the limited technology, the teacher needed to make 
adjustments to the small group portion of the curriculum. To allow students ample time on the 
computers, the teacher developed additional centers dedicated to the mathematics skills cov¬ 
ered in the episode. For example, "Spooky Summer Soiree" focuses on money. In addition to the 
worksheet, Word Wall activity and Prankster Planet, the teacher created two additional centers. 
He developed a memory card game using various amounts of money that a student needed to 
match. Another game uses plastic money to ‘buy’ various items listed in the coupon section of 
the newspaper. The teacher commented that the group of students was particularly active, and 
the TEC curriculum kept these students engaged: “I could see that they were totally ready. They 
were totally engaged. They wanted to do it. They wanted to dance around and to move. That’s 
what they want and need. For the boys, it was great. They were loving it.” 


MEASURES 

Each research question was addressed using multiple measures. Summer school program 
characteristics, TEC curriculum implementation, and teacher responses (e.g., their reactions to 
the curriculum) were measured through teacher surveys and interviews. Student outcomes were 
measured through student surveys, student interviews, and teacher surveys. Exhibit 2 illustrates 
how measures align with the study’s research questions. 
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Exhibit 2: Alignment Between Research Questions and Measures 


Research Questions 

Measures 

1. Are teachers implementing the TEC Summer 
Learning Program as planned? Are there any 
obstacles to implementation? What variables 
contribute to fidelity of implementation? (e.g., 
number of children, children’s initial level of 
knowledge, years of teacher’s experience, type 
of program). 

Teacher Intake Survey 
Teacher Exit Survey 

Teacher Interview 

Classroom Observation 

Prankster Planet Student 
Usage Data 

2. Do students who participate in the TEC Summer 
Learning Program increase skills in select 
mathematical domains? 

Standards/Skills Aligned 
Survey (Pre-Post) 

Teacher Exit Survey 

Teacher Interview 

3. Do students who participate in the TEC Summer 
Learning Program acquire the mathematics 
vocabulary targeted by the program? 

Standards/Skills Aligned 
Survey (Pre-Post) 

Student Interview 

Teacher Exit Survey 

Teacher Interview 

Prankster Planet Student 
Usage Data 

4. Do students who participate in the TEC 

Summer Learning Program increase in their 
comprehension of connected text? Do they learn 
strategies that good readers use to understand 
connected text? 

Standards/Skills Aligned 
Survey (Pre-Post) 

Teacher Exit Survey 

Teacher Interview 

5. Do students who participate in the TEC Summer 
Learning Program increase in their motivation 
and confidence to engaging in learning activities 
related to literacy? 

Standards/Skills Aligned 
Survey (Pre-Post) 

Student Interview 

Teacher Exit Survey 

Teacher Interview 

6. Do students who participate in the TEC Summer 
Learning Program increase in their motivation 
and confidence to engage in learning activities 
related to mathematics? 

Standards/Skills Aligned 
Survey (Pre-Post) 

Student Interview 

Teacher Exit Survey 

Teacher Interview 

7. Do teachers who participate in the TEC Summer 
Learning Program increase in their motivation 
and confidence leading groups? 

Teacher Exit Survey 

Teacher Interview 

8. Do teachers who participate in the TEC Summer 
Learning Program report an increase in morale 
and enthusiasm in their summer school setting? 

Teacher Exit Survey 

Teacher Interview 
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Research Questions 


Measures 


9. Do teachers who participate in the TEC Summer 
Learning Program learn new instructional 
strategies in teaching mathematics and literacy? 

Teacher Exit Survey 

Teacher Interview 

10.Which aspects of the TEC model do children 
find the most appealing, interesting, and 
educational? 

Student Interview 

Teacher Exit Survey 

Teacher Interview 

Classroom Observation 

Prankster Planet Student 
Usage Data 

11. Which aspects of the TEC model do teachers 
find the most appealing, interesting, and 
valuable? 

Teacher Exit Survey 

Teacher Interview 

Prankster Planet Student 
Usage Data 

12. Which aspects, if any, of the TEC model 
(including professional development) do 
the teachers think might need to be refined, 
improved, or modified? Would teachers use the 
summer curriculum again? Why or why not? 

Teacher Exit Survey 

Teacher Interview 


Each of the measures is discussed in more detail below. 


Standards/Skills Aligned Survey 

WestEd developed a multiple-choice standards/skills aligned survey using released items from 
the mathematics and English-language arts California Standards Tests. The survey was group 
administered by participating teachers in the study during week one of their program as a 
pre-test and during the final week of their program as a post-test. The 19-item survey contained 
items aligned with the TEC curriculum. 3 The 17 mathematics items dealt with knowledge of 
place value, addition, subtraction, measurement, data analysis, and time. The two phonics items 
dealt with letter-sound knowledge. The students' scores on the survey were based on the total 
number of correctly completed items. 

We assessed the reliability of the survey using the split-half method. In the split-half method, the 
students' scores from the even-numbered items are correlated with their scores from the odd- 
numbered items (Hopkins, 1998). We utilized the Spearman-Brown formula to calculate the split- 
half reliability of the survey. The survey showed acceptable reliability at the pre-test (pxx = .72) 
and the post-test (pxx = .78). 


3 The post-test version of the survey contained several additional items that were not included in the 
data analysis for the formative study. 
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In order to assess the students’ attitudes about reading and mathematics, researchers utilized a 
10-item self-report scale that was based on the Estes Attitudinal Scale (Estes, Estes, & Richards, 
1985). The reading subscale contained five items that assessed the students’ attitudes about 
reading (e.g., There should be more time for reading during the school day) and the math¬ 
ematics subscale included five of items that assessed the students’ attitudes about mathematics 
(e.g., It is easy to understand mathematics). The response options for the items were: / disagree, 

/ agree, or / don’t know. Students circled their response after each item was read to the students 
by the teacher. 

We evaluated the reliability of the subscales using Cronbach’s alpha, which is a measure of 
internal consistency reliability. A subscale with high levels of internal consistency contains items 
that are most likely measuring the same construct. In general, Cronbach’s alpha values above 
.70 are considered acceptable (Cortina, 1993). The attitudes about reading subscale did not 
show acceptable reliability at either the pre-test (a = .32) or the post-test (a = .61). Similarly, 
the attitudes about mathematics subscale did not have acceptable reliability at the pre-test 
(a = .54) and the post-test (a = .52). As a result of these low reliabilities, we did not evaluate the 
students’ growth on the measures. In the Findings section, we present only the frequencies for 
the post-test. 


Student Interview 

Researchers created both pre and post student interview protocols. Both protocols included 
a mathematics vocabulary section that required the students to verbally provide definition of 
20 mathematics-related words (e.g., quarter, graph) that were covered by the TEC curriculum. 
WestEd staff interviewed the students individually during week one of their program as a pre¬ 
test, and during their final week as a post-test. The interviewers scored the items as correct, 
incorrect, or partially correct. For the current analysis, the students received credit only for cor¬ 
rect responses and the students’ scores were based on the number of correctly defined words. 
Consistent with the standards/skills aligned survey, we utilized the Spearman-Brown formula to 
assess the split-half reliability of the vocabulary section of the interview. The vocabulary section 
had acceptable reliability at the pre-test (pxx = .75) and the post-test (pxx = .75). Prior to the 
mathematics vocabulary section, the interviewers posed several open-ended questions to the 
students. For the pre-interview, the questions related to what the students liked to do in school 
and what kind of computer games they liked to play. For the post-interview, the students were 
asked about their experiences with the TEC curriculum, including what they liked best about it. 


Prankster Planet Usage Data 

Individual student website usage click data and Flash tags were collected as the participating 
students used Prankster Planet. Different levels of the game corresponded to the 12 episodes of 
The Electric Company DVDs. For each level, both the number of times a student entered and the 
number of times the level was completed were tracked. Also tracked were the number of tries 
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it took students to complete the Vocabulary Challenges within each level. The usage data was 
collected from Google Analytics and was measured in hours. 

Students actively used the Prankster Planet website at their summer school programs and much 
of their website usage was tracked in Google Analytics. However, not all usage was tracked due 
to the following issues: 

» Some teachers received instructions on how to log-in to Prankster Planet through the 
research log-in site after they had already begun their program and some teachers were 
confused about logging in, because the curriculum guide pointed to a different log-in 
site (pbskids.org/electriccompany/pranksterplanet). 

» Students would sometimes access Prankster Planet through the PBS KIDS GO! website 
and log-in through the main URL (pbskids.org/go), rather than through the research 
log-in site (pbskids.org/wested). Thus, students were able to log-in and access the PBS 
KIDS GO! site, but their usage was not captured through Google Analytics. 

» Students had difficulty remembering their usernames, passwords, and secret codes (for 
password reminders). In addition, some students had difficulty logging in because they 
were not yet adept with the keyboard and typing. This posed a logistical problem for 
teachers, as they had to log-in each individual student themselves, before students could 
begin using the website. 

» Student username rosters were not always accurate. Students would often forget their 
usernames and would create new usernames to access the site. However, the new user- 
names would often not be reported back to WestEd. 

» Several teachers reported that some students were unable to log-in using their correct 
username. They would get a message stating that they could not use the username, so 
they would make up a new username each time they logged on. 

Each of these issues was identified and addressed throughout the study. 


Teacher Intake Survey 

The Teacher Intake Survey was developed to determine possible participants for the study and 
to gather background data on teachers. The survey addresses teacher demographics, teacher 
experience, and school and classroom variables. Teachers were interviewed by phone if their 
survey results met the study criteria. 


Classroom Observation 

A mid-curriculum visit took place at 9 of the 12 sites in order to track fidelity of implementa¬ 
tion. WestEd created an observation protocol where researchers kept a running record of the 
90-minute lesson presented that day, noting in particular: modifications/alignment with the TEC 
Summer Learning Program curriculum, student engagement, student learning, and technical 
difficulties. 
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Teacher Interview 

Researchers conducted a mid-visit interview with 13 of the 16 participating TEC teachers, and a 
final interview with every participating teacher. WestEd developed the interview questions to 
both address the research questions and gain feedback on the TEC Summer Learning Program 
curriculum. The mid-visit interview, done in conjunction with the classroom observation, specifi¬ 
cally addressed the observed lesson. The final interview addressed the curriculum as a whole. 


Teacher Exit Survey 

WestEd created the teacher exit survey to address both quantitative and qualitative findings 
with respect to the research questions. The survey addressed characteristics of the program and 
students, teacher perceptions of student learning, motivation, and confidence, and feedback on 
the TEC Summer Learning Program. 


DATA ANALYSIS 

To investigate the association between the TEC Summer Learning Program and participating stu¬ 
dents' knowledge and vocabulary, we utilized a one-group pre-test/post-test design (Shadish, 
Cook, & Campbell, 2002) and, therefore analyzed the data using a series of single-subject 
t-tests. Descriptive statistics, such as frequencies, means, standard deviations, and ranges, were 
used to answer the research questions about program implementation and teacher outcomes. 
The student interview data, that were used to supplement the findings from the student survey 
and vocabulary section of the interview, were transcribed and coded using the Atlas.ti analysis 
program. All teacher interviews were transcribed and coded using the HyperResearch software 
program. Coded transcripts were analyzed for themes that informed the answers to each rel¬ 
evant research question. 

There are missing student data for the study because of one or more of the following reasons: 
student absences on pre-post test/interview dates, late enrollment of students, and/or consent 
forms not received until partial completion of the summer program. Only complete cases were 
used in the pre/post analyses. 


( 


WestEd // Evaluation of The Electric Company Summer Learning Program // INTRODUCTION 


page 


23 





FINDINGS 


J 


The following section presents findings from analyses that address each of the research ques¬ 
tions concerning implementation of the TEC Summer Learning Program , student outcomes, 
teacher outcomes, teacher feedback, and suggestions for improvement. The section begins 
with findings on the extent to which the summer school programs in the study implemented the 
TEC Summer Learning Program as planned as well as the obstacles to implementation. In this 
section, we report on what variables contributed to fidelity of implementation with a summary 
of the characteristics of the summer school programs within which TEC was delivered, as well 
as characteristics of those programs’ teachers and students. The next section discusses student 
outcomes both before and after the program was implemented. This includes findings from 
analysis of the student surveys, as well as from the student interviews and teacher surveys. The 
next section reports the findings on teacher outcomes, including the extent to which the TEC 
Summer Learning Program increased their motivation and confidence, and learned new instruc¬ 
tional strategies. The fourth section reports what teachers found to be most and least appealing, 
interesting, and valuable from both the students' perspectives and their own. This section also 
reports teachers’ interest in using the curriculum again, and how they would do so. The final 
section reports on which aspects of the TEC Summer Learning Program teachers think could 
be refined, improved, or modified. This section provides suggestions for both the professional 
development training and the curriculum. 


PROGRAM IMPLEMENTATION 

This section addresses the research questions that pertain to implementation of the TEC 
Summer Learning Program curriculum. These analyses used data from both teacher interviews 
and the teacher exit survey as well as classroom observations. The section begins with a discus¬ 
sion of the characteristics of the summer programs in which the TEC curriculum was imple¬ 
mented (and characteristics of the teachers and enrolled students). 

Teacher Experience 

Exhibit 3 shows that the majority of the summer school teachers in the study were relatively new 
to teaching. Half of the 16 teachers reported that they had been teaching for five or fewer years, 
three reported they had been teaching 6 to 10 years, two reported that they been teaching 11 to 
15 years, and three reported 15 or more years. 
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Exhibit 3: Number of Years Teaching as Reported by Summer School Program 
Teachers (n = 16) 



5 or fewer 6 to 10 11 to 15 More than 15 


Academic Rigor of Summer School Programs 

Teachers were also asked about the perceived rigor of the mathematics and English language 
arts content of their summer school programs on a scale from “not particularly rigorous” to 
“extremely rigorous.” The findings from this question appear in Exhibit 4. Six teachers reported 
that the programs were "somewhat rigorous” with respect to mathematics followed by four 
teachers who said their programs were “very rigorous.” None of the teachers reported that their 
program was "extremely rigorous” with regards to mathematics. The results were similar for 
teachers’ perceived rigor of the English language arts portions of the program with 6 reporting 
that their programs were “somewhat rigorous” followed by 5 who responded that their pro¬ 
grams were “very rigorous.” Two teachers responded that the English language arts aspects of 
their programs were “extremely rigorous.” 
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Exhibit 4: Teacher Rating of Summer School Programs' Academic Rigor (n = 16) 


<D 
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No focus 


Not 


on subject particularly 
area rigorous 


Somewhat 

rigorous 


Very 

rigorous 


Extremely 

rigorous 


Student Enrollment 

There was variability in the number of students enrolled in the summer school programs that 
implemented the TEC curriculum (Exhibit 5). Teachers at five of 16 sites reported that the size 
of their summer school class was between 11 to 15 students and another five teachers reported 
class sizes of 16 to 20 students. One class had fewer than five students enrolled and one had 
more than 15. In addition, a large number of the teachers estimated that relatively few students 
in their summer school classes were Caucasian (Exhibit 6). Specifically, teachers in 10 of the 
16 classrooms estimated that the percentage of Caucasian students was less than 10 percent, 
and four teachers reported that the percentage of Caucasian students was 10 or 20 percent. 
Finally, the majority of teachers estimated either that there were no students who were Limited 
English Proficient (LEP) in their summer school classrooms, or that there were fewer than 10 per¬ 
cent (Exhibit 7). 
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Exhibit 5: Number of Students Enrolled in Summer School Programs (n = 16) 

io - 

9 - 

i/i 8 

E 

2 7 - 

U) 

O 6 - 

CL 


CD 4 
_Q 

E 3 

D 

z 2 
1 

0 
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Exhibit 6: Teacher Estimated Percentage of Caucasian Students in the Summer 
School Programs (n = 76) 



10% or 20% or 30% 40% or 50% 60% or 70% 80% or more 
fewer 
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Exhibit 7: Percentage of Students with Limited English Proficiency in Summer School 
Classrooms (n = 16) 
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None or less 10% to 25% 26% to 50% More than 50% 

than 10% 


ARE TEACHERS IMPLEMENTING THE TEC SUMMER LEARNING 
PROGRAM AS PLANNED? WHAT VARIABLES CONTRIBUTE TO 
FIDELITY OF IMPLEMENTATION? 

WestEd found that though the summer programs enrolled in the TEC Summer Learning Program 
study were extremely diverse, teachers primarily implemented the curriculum as planned, with 
some expected modifications. This section describes two aspects of implementation: training of 
teachers and delivery of episodes. Even with variation in these two aspects of implementation, 
student outcomes were not negatively affected. 

Summer school teachers were offered four possible times (two per part) to attend Parts 1 
and 2 of a live orientation webinar to prepare them for implementing the TEC Summer Learning 
Program curriculum as intended. In addition, teachers could view archived versions of both parts 
of the orientation webinar at a later date for the purposes of either orientating themselves to 
the curriculum and its implementation or refreshing their memories. Seven of the 16 teachers 
reported that they attended both portions of the live orientation webinar (Exhibit 8). Three 
teachers reported that they combined watching one archived webinar while attending the 
other part in the live version. Two teachers reported that they received orientation through 
the archived webinar only . Finally, two teachers reported that they attended both the live and 
archived formats, and two teachers reported that they attended neither the live nor the archived 
formats. 
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Exhibit 8: Number of Teachers Who Attended Webinar Orientation Formats (n = 16) 
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The two teachers who did not attend either formats of the orientation webinar veered the 
farthest from the intended implementation of the TEC Summer Learning Program curriculum. 
One of these teachers began her summer program by showing Sessions 1 and 2 of the same 
episode on the same day because she was worried she would run out of time. She also had 
technical difficulties that made it impossible to “pause and play” as intended with Session 2 
of the episode. This teacher also made the time allotted for playing Prankster Planet on the 
computers optional, as this time was attached to the students’ lunch and recess. She reported 
that the boys seemed more interested than the girls in coming in during their “free time” to play 
Prankster Planet. These issues came up during the mid-program classroom observation, where 
the visiting researcher was able to guide the teacher toward the intended implementation of 
the curriculum for the rest of her summer program. The other teacher who did not attend either 
format of the orientation webinar began to skip days of the curriculum toward the end of her 
summer program. Sometimes she chose not to watch an episode a second time, and just move 
ahead to the activities. 

We also examined the number of summer school programs that delivered each of the 12 TEC 
episodes. All summer school programs, with the exception of one, viewed at least ten of the TEC 
episodes. Specifically, six programs viewed all 12 episodes, two programs viewed 11 episodes, and 
seven programs viewed ten. One summer program viewed only eight episodes due to disruptions 
at the school site. According to the teacher survey, episodes were skipped where the summer 
sessions were less than six weeks. Episodes 3 (“The Incredible Return-A-Ball) and 5 (“Wiki Wiki 
Walter”) were designated by Sesame Workshop as the episodes to skip, if absolutely necessary. 
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Most teachers implemented the TEC Summer Learning Program as planned, making modifica¬ 
tions such as changing the order of some activities, or changing the types of groupings for 
certain activities based on the structure of their summer program. Many teachers added addi¬ 
tional literacy and mathematics activities to the curriculum, as they felt they were needed for 
their particular programs. 


ARE THERE ANY OBSTACLES TO IMPLEMENTATION? 

When teachers were asked about obstacles to implementing the TEC curriculum on the exit 
survey, three of the 16 teachers responded that there were none. Of the teachers who did note 
obstacles, three mentioned that either students had dropped out of the program or had trouble 
attending (e.g., due to transportation issues). Three teachers mentioned they experienced 
technical difficulties, specifically with computers to run Prankster Planet, malfunctioning copy 
machines, or lack of access to a DVD player on a particular day. Several teachers remarked in 
interviews that they didn’t have enough computers in their room to work the curriculum effec¬ 
tively as planned. One teacher, in particular, felt having two students per computer simply didn’t 
work in her classroom. 

Three other types of obstacles to implementation came up during teacher interviews: student 
knowledge of technology, the summer program structure, and teacher issues with the TEC 
Summer Learning Program curriculum. First, several teachers reported that their students’ lack 
of knowledge with respect to technology proved to be an obstacle. 

And the computer part of it, / had no idea! Some of them actually went In there, not 
knowing how to use the computer...you know, throwing out words like "mouse” and 
"space bar" and "left-right arrow." They had no idea what they meant. 

Especially with the population that / teach, some of these kids are not very well exposed 
to computers. So that creates some problems. 

Some teachers found that the structures of their particular summer programs made it chal¬ 
lenging to implement the TEC curriculum as planned. For one site, the computer lab was a 
ten-minute walk from the classroom, and they could only access it during a certain time frame. 
They had to take time from other activities in order to be able to include time on the computer. 
Another teacher mentioned that she often felt pressured by only having 90 minutes a day with 
her students to do the curriculum. She felt the curriculum deserved more than 90 minutes, but 
in her program that is all she was allotted. Other teachers remarked that the size of their classes 
made in difficult to implement the curriculum as planned. One teacher, who had only five stu¬ 
dents in her class, found that the estimated times in the curriculum often didn’t work for her: 

The only other thing / might have adjusted was sometimes the 25 minutes for the group 
activities was kind of hard for me to stretch that long with the three to five kids, so / 
would either add in a little bit to keep that 25 minute block or we'd have to end, you 
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know, like cut it five minutes shorter or something like that, just because, if it got kind of 
too repetitive for them, it was kind of an overkill for them. 

Another teacher, who had a large class of 25 students, remarked: 

/ think / was able to handle it...but / think that it is for a small class. Because it's pretty dif¬ 
ficult to help the kids with the worksheet and also be available for those who need some¬ 
thing with the computers. Something, either their Internet connection failed or they're on 
the wrong web site. 

The third obstacle to implementation that came out in teacher interviews was teacher issues 
with the curriculum. Some teachers referenced the time allotted to certain activities—two 
teachers reported that the time for small group activities was too short, and two teachers men¬ 
tioned that the large group activities took longer than estimated in the instructional materials. 
Other teachers mentioned problems with student readiness for the content of the curriculum: 

I’m so grateful to be here. But you know, they needed—and even with the game, a lot of 
the games it was adding numbers over a hundred. And some of them just aren’t ready 
for that yet, and they needed a lot of support. You know, some of the games they weren’t 
able to complete independently, because, again, it focused more on skills that are learned 
in second grade versus reinforcing the skills that they learned in first grade. 

It was the money. When we were doing the money and subtracting and adding, and I’d 
go, okay, we’re going to go buy this. How much is it? For them to add decimals, it was like 
to add 25 cents to 10 cents to five cents to a dollar, that was just so foreign to them. / just 
was just, okay, hold on. Do you guys know how to do money? They were like, not really, 
kind of. And a couple of them did and a couple of them didn’t. But there was such a big 
gap there that they were lost. 

The only thing that they have trouble with is the money piece...They have never been 
introduced to that, which is part of where we come from, like the schools, but they had no 
idea. 

But a lot of these, the Word Wall group activities, I feel as if they require a lot of modeling 
in order for them to be done well, and actually get anything out of it. So / didn’t get to a 
lot of those activities. 

Sometimes pieces were challenging for them in this grade, because of limited reading 
ability, or their reading levels. Sometimes the session two worksheets...that was some¬ 
times challenging...And sometimes / would have to read the captions for them, so they 
would know what to do—what was being looked for. 
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STUDENT OUTCOMES 

In this section we address the research questions that concern student outcomes. These 
analyses used data from the student standards/skills aligned survey, the student interview, and 
the teacher exit survey. First, we report the association between all the learning outcomes and 
attitudinal measures. Next, we discuss whether students who participated in the TEC Summer 
Learning Program exhibit an increase in learning outcomes, specifically increased knowledge 
of phonics and in mathematical domains, and whether they acquire the specific vocabulary 
targeted by the program. We also discuss the association between these learning outcomes 
and one aspect of the TEC model—usage of Prankster Planet. In addition, this section examines 
students' attitudes towards mathematics and reading. 

In order to understand the association between attitudes and outcomes on student math¬ 
ematics, phonics, and verbal surveys, WestEd conducted a series of Pearson-r correlations 
between all the student outcome measures at post-test. The results appear in Exhibit 9. There 
was a statistically significant positive correlation between students' mathematics skills, phonics 
skills, and mathematics vocabulary, indicating that when students did well on one portion of the 
outcome assessments they also tended to do well on all the other portions of the assessments. 

In addition, there was a statistically significant positive correlation between students' attitudes 
about reading and attitudes about mathematics. Attitudes about mathematics were significantly 
correlated with mathematics survey items. However, there was no statistically significant correla¬ 
tion between attitudes about reading and any group of mathematics survey items or the math¬ 
ematics vocabulary section of the student interview. Thus, more positive attitudes about math 
were associated with better performance on the assessments while attitudes about reading were 
associated with neither better nor worse performance. 


( 


WestEd // Evaluation of The Electric Company Summer Learning Program // FINDINGS 


page 


32 





Exhibit 9: Intercorrelations among the Post-Test Measures 
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1. Standards/Skills 
Aligned Survey 
(All Items) 

- 






2. Mathematics 

Survey Items Only 

98*** 

- 





3. Phonics Survey 

Items Only 

.64*** 

.48*** 

- 




4. Mathematics 

Vocabulary Section 
of Interview 

.56*** 

.53*** 

.43*** 

- 



5. Attitudes about 
Reading 

.12 

.14 

-.03 

.03 

- 


6. Attitudes about 
Mathematics 

.18* 

.20* 

.04 

.07 

37*** 

- 


Note: The ns range from 101 to 126 for the correlations. *p < .05. ***p < .001. 


DO STUDENTS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM INCREASE SKILLS IN SELECT MATHEMATICAL DOMAINS? 

In order to assess whether the students who participated in the TEC Summer Learning Program 
increased their skills in select mathematical domains, we compared their average pre-test scores 
on the standards/skills aligned survey with their average post-test scores. The students’ mean 
scores on the full 19-item survey and the 17 mathematics items are shown in Exhibit 10. On the 
full survey, the students who participated in the program successfully completed nearly two 
items more at the post-test than they did at the pre-test. Specifically, the students' scores on 
the full survey increased from a mean of 8.88 (SD = 3.34) to a mean of 10.63 (SD = 3.73), which 
translated to a 20% gain and was statistically significant, f(111) = 5.15, p < .001. In addition, the 
students' scores showed 20% growth on the mathematics items, which was also statistically 
significant, f(111) = 5.02, p < .001. We translated the pre- and post-test means into standardized 
effect sizes to help interpret the magnitude of the students’ growth. The effect sizes were esti¬ 
mated by calculating the difference between the pre- and post-tests means and dividing the dif¬ 
ference scores by the pooled standard deviations for the pre and post tests (Hill, Bloom, Black, & 
Lipsey, 2008). For the full survey and the mathematics questions, the effect sizes were d = 0.50 
and d = 0.49, respectively. These effect sizes are considered medium sized based on Cohen’s 
(1988) guidelines for interpreting effect sizes. However, it should be noted that these medium 
effect sizes occurred after only six weeks, at most, of the TEC Summer Learning Program. 
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Exhibit 10: Mathematics Survey Scores at the Pre-Test and Post-Test 



Pre-Test 

Post- 

Test 













Mean 

SD 

Mean 

SD 

Difference 

t 

Cohen’s 

d 

P 

Full Survey 
(19 Items) 

8.88 

3.34 

10.63 

3.73 

1.76 

5.15 

0.50 

<.001 

Mathematics 

Items Only 
(17 Items) 

7.83 

3.08 

9.41 

3.28 

1.58 

5.02 

0.49 

<.001 


Note: Mean scores are based on the number of correctly completed items. N = 112. 


The percentage of students who correctly answered each of the 19 items on the standards/skills 
aligned survey at the pre- and post-test is included in Exhibit 11. The items on the survey varied 
in difficulty. At the pre-test, for example, 78.6% (n = 88) of the students correctly completed 
item 19 while only 19.6% (n = 22) of the students correctly responded to item 1. For 15 of the 
items, the percentage of students who provided correct responses increased from the pre-test 
to the post-test. In contrast, the percentage of students who correctly completed the other four 
items either declined or stayed the same from pre- to post-test. 


Exhibit 11: Percent Correct for Items on the Standards/Skills Aligned Survey at the 
Pre-Test and Post-Test 



Pre-Test: 

Correct 

Responses 

Post-Test: 

Correct 

Responses 

Diffe 

betw 

Pre a 

Post 1 

rence 

een 

nd 

Item 

% 

n 

% 

n 



1. A number has nine ones, six tens, and 
eight hundreds. What is the number? 

19.6% 

22 

28.6% 

32 

9.0% 

2. Look at the number (962). Which digit is in the 
tens place? 

33.9% 

38 

41.1% 

46 

7.2% 

3. Which digit is in the ones place in the number 
two hundred thirty-four? (234) 

33.9% 

38 

33.0% 

37 

-0.9% 

4. What is another way to write nine hundred 
eighty-seven? 

34.8% 

39 

61.6% 

69 

26.8% 

5. Look at the two problems in the box. (65 - x = 60; 
60 + x = 65) The same number is missing in both of 
them. What is the missing number? 

58.0% 

65 

64.3% 

72 

6.3% 
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Pre-Test: 

Correct 

Responses 

Post-Test: 

Correct 

Responses 

Difference 

between 

Pre and 

Post 1 

Item 

% 

n 

% 

- 


6. What is the solution to this problem? (410 + 94) 

35.7% 

40 

29.5% 

33 

-6.2% 

7. Monique has four quarters, two dimes, and one 
nickel. How much money does she have? 

48.2% 

54 

58.0% 

65 

9.8% 

8. Jena has the amount of money you see in the box. 
Which is a greater amount of money than Jena’s? 

33.9% 

38 

36.6% 

41 

2.7% 

9. Lee has the money you see in the box. How much 
money is this? 

39.3% 

44 

52.7% 

59 

13.4% 

10. About how long is a dollar bill? 

40.2% 

45 

56.3% 

63 

16.1% 

11. About how many jellybeans long is the pencil? 

50.9% 

57 

67.0% 

75 

16.1% 

12. This comb is about 12 buttons long. About how 
many toothpicks long is the comb? 

47.3% 

53 

65.2% 

73 

17.9% 

13. Natalie walked for one hour. How many minutes 
did Natalie walk? 

55.4% 

62 

44.6% 

50 

-10.8% 

14. Which tally chart shows the correct number of 
pets in Sam’s pet shop? 

55.4% 

62 

72.3% 

81 

16.9% 

15. Look at the tally chart at the top of the page. The 
tally chart shows the number of cherries each student 
ate. Which graph matches the tally marks in the chart? 

56.3% 

63 

71.4% 

80 

15.1% 

16. The bar graph shows a favorite flavor of juice 
for a group of people. Which of the following tally 
charts matches the bar graph? 

61.6% 

69 

80.4% 

90 

18.8% 

17. Find the word that has the same sound as 

the underlined letter or letters in the first word. 
(Elephant) 

44.6% 

50 

51.8% 

58 

7.2% 

18. Find the word that has the same sound as the 
underlined letter or letters in the first word. (Meet) 

59.8% 

67 

70.5% 

79 

10.7% 

19. What time does the clock read? 

78.6% 

88 

78.6% 

88 

0.0% 


Note: N = 112; 1 = Positive differences indicate gains from pre- to post-assessment 

Teachers reported many noticeable student gains in select mathematical domains, such as: addi¬ 
tion and subtraction, place value, counting money, telling time, measuring, and balancing a scale. 
Specific comments include: 
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/ mean these kids came in, like / said, not even knowing how to add. And they're on the 
computer now doing a thousand plus two hundred, and two hundred, a thousand, four 
hundred. And they're doing it by themselves. 

The zero one too is pretty cool. You know, is forty more than four hundred, or less than 
four hundred? They were able to tell you, you know. Several of them didn’t really know. 
Would you rather have forty dollars or would you rather have four hundred dollars? Some 
of them didn't know. But after we played the game, and we saw where zero held the place 
and what's zero’s place value, / think they walked away with that. 

A lot of—the money thing that was really hard, but / really think that they got, like the 
estimation part of it. They really started to understand, which was kind of exciting. Like, 
you have five dollars to spend. And the pigtail costume...and they were adding it up. Do 
you have enough money to buy all of it?...And most of them were able to figure out, yeah, 

/ know / don’t have enough money, or yes, / can buy all those things. That was kind of 
exciting. When you’re teaching first grade, the concept of estimating how much, it's just 
so hard and really hard to conceptualize. That was one task that they really came away 
with strong after that. 

You hear it in their conversations, or if we tell them to look at the dock and tell us what 
time it is, they can do that. 

There’s a couple that they really liked. Oh, the balancing ones. When we moved from 
one side to the other, the other side of the room, they really liked that one. And it really 
helped. Like if you have more people over here, and you don’t have as many people over 
here, how are we going to balance that out? And they would kind of start seeing, okay, 
this is equal. 

One teacher, in particular, commented extensively on how the TEC Summer Learning Program 
curriculum made the mathematics content more accessible and motivating for his students to 
learn: 


/ would say that learning money through watching the show and then trying to do it on 
Prankster Planet opens like an avenue for them. It's like something in their brain. It opens 
a little pathway of like / want to learn it. Versus if I just sat there, this is a quarter, this is a 
dime...So all those regular classroom activities are a lot more accessible now that we’ve 
been able to kind of open up their minds through the TV and through the computer. 


DO STUDENTS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM ACQUIRE THE MATHEMATICS VOCABULARY TARGETED BY 
THE PROGRAM? 

To determine whether the students who participated in the TEC Summer Learning Program 
acquired the mathematics vocabulary targeted by the program, we compared their mean 
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pre-test scores on the mathematics vocabulary section of the student interview with their mean 
post-test scores (Exhibit 12). The students correctly defined an average of 3.30 more words at 
the post-test than they did at the pre-test. Specifically, the students’ scores increased signifi¬ 
cantly, f(81) = 12.33, p < .001, from an average of 8.10 (SD = 3.27) to an average of 11.40 (SD = 
3.58). The effect size that characterized the students’ growth from pre- to post-test (d = 0.96) 
is considered a large effect size (Cohen, 1988) and represents a gain of 41% from pre- to post¬ 
assessment. This is particularly notable given that the time between the tests was only six 
weeks, at most. 


Exhibit 12: Mathematics Vocabulary Scores at the Pre-Test and Post-Test 



Pre-Test 

Post-Test 
















Mean 

SD 

Mean 

S 

D 

Difference 

t 

Cohen’s 

d 


Full Test (20 Items) 

8.10 

3.27 

11.40 

3.58 

3.30 

12.33 

0.96 

<.001 


Note: Mean scores are based on the number of correctly defined words. N = 82. 


The percentage of students who correctly defined each of the 20 words on the mathematics 
vocabulary section of the student interview at the pre- and post-test is outlined in Exhibit 13. 

The difficulty of the words varied significantly. For example, at the pre-test, 90.2% (n - 74) of 
the students correctly defined “straight” and just 2.4% (n - 2) of the students correctly defined 
“likely”. For all 20 words on the vocabulary test, the percentage of students who provided a cor¬ 
rect definition increased from the pre-test to the post-test. 
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Exhibit 13: Percent Correct for Items on the Mathematics Vocabulary Section of the 
Student Interview at the Pre-Test and Post-Test 



Pre-Test: 

Correct 

Responses 

Post-Test: 

Correct 

Responses 

Difference 

between 

Pre and Post 1 

Vocabulary Word 

% 

n 

% 

" 


1. Target 

51.2% 

42 

64.6% 

53 

13.4% 

2. Straight 

90.2% 

74 

97.6% 

80 

7.4% 

3. Batch 

9.8% 

8 

35.4% 

29 

25.6% 

4. Equal 

67.1% 

55 

78.0% 

64 

10.9% 

5. Fives 

82.9% 

68 

95.1% 

78 

12.2% 

6. Graph 

19.5% 

16 

35.4% 

29 

15.9% 

7. Goal 

11.0% 

9 

25.6% 

21 

14.6% 

8. Inch 

23.2% 

19 

72.0% 

59 

48.8% 

9. Shrink 

58.5% 

48 

87.8% 

72 

29.3% 

10. Dollar 

18.3% 

15 

47.6% 

39 

29.3% 

11. Quarter 

61.0% 

50 

67.1% 

55 

6.1% 

12. Dime 

67.1% 

55 

74.4% 

61 

7.3% 

13. Amount 

17.1% 

14 

36.6% 

30 

19.5% 

14. Zero 

87.8% 

72 

96.3% 

79 

8.5% 

15. Scale 

25.6% 

21 

41.5% 

34 

15.9% 

16. Tens 

87.8% 

72 

96.3% 

79 

8.5% 

17. Minute 

12.2% 

10 

31.7% 

26 

19.5% 

18. Hour 

12.2% 

10 

23.2% 

19 

11.0% 

19. Likely 

2.4% 

2 

13.4% 

11 

11.0% 

20. Inflate 

4.9% 

4 

20.7% 

17 

15.8% 


Note: N = 82, 1 =Positive differences indicate gains from pre- to post-assessment 

Many teachers reported that they were impressed with the vocabulary gains their students 
made. Comments include: 
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/ do see some improvement especially in the vocabulary. / think they do learn how to 
use these words in their vocabulary. And / know that some of these students when they 
respond in class and they use the vocabulary. 

So the structure of the five electric words in the beginning, and powering up during them, 
and have them repeatedly show up both in Prankster Planet and on the worksheets. I 
think that’s pretty strong. So / would say the vocabulary has gotten a lot stronger. And to 
identify those words and to know what they mean. 

The language is incredible, because that's another issue. You know, they don’t have high 
vocabulary. Having the discussions about the vocabulary words is so beneficial. Even today 
we talked about homonyms, because when / showed them weight, and they’re like, oh, it 
means to take a turn. Takes turn. I'm like, whoa! It’s a good teachable moment for them. 

/ would say the most valuable for them was the vocabulary words and the word wall. / 
definitely think that was really good. Because they are words that they are going to use 
throughout the year that they now have a better understanding going in. 


DO STUDENTS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM INCREASE COMPREHENSION OF CONNECTED TEXT? 

DO THEY LEARN STRATEGIES THAT GOOD READERS USE TO 
UNDERSTAND CONNECTED TEXT? 

At the end of their summer programs, teachers were asked if they felt their students had pro¬ 
gressed in their comprehension of connected text. Eleven out of 16 teachers reported that their 
students did progress in this area. Twelve out of 16 teachers reported that students had pro¬ 
gressed in the area of “learned and used strategies to understand connected text.” 

In addition, Exhibit 14 shows that students did show 17% growth on the two phonics items of the 
Standards/Skills Aligned Survey, f(111) = 2.07, p < .05. 


Exhibit 14: Phonics Items from the Student Survey Scores at the Pre-Test and 
Post-Test 



Pre-Test 

Post-Test 
















Mean 

SD 

Mean 


SD 

Difference 

t 

Cohen’s 

d 

P 

Full Survey (19 Items) 

8.88 

3.34 

10.63 

3.73 

1.76 

5.15 

0.50 

<.001 

Phonics Items Only 
(2 Items) 

1.04 

0.79 

1.22 

0.77 

0.18 

2.07 

0.23 

<.05 


Note: Mean scores are based on the number of correctly completed items. N = 112. 


c 


WestEd // Evaluation of The Electric Company Summer Learning Program // FINDINGS 


page 


39 



























Many teachers reported that they felt the TEC curriculum had positively reinforced various 
reading strategies in their students. Specific comments include: 

/ think they are learning how to use words. Especially the part where it says "read through 
to the end." Some of the students, they’ll see a multisyllabic word and or it's too long and 
sometimes are just going to say anything. Well, now, having the opportunity to look at 
how the Electric Company characters break the words down into manageable chunks, 
and then pronounce the words, is helping them. 

it is a work in progress here, because these aren’t my regular students, but i do know 
that some of them still have trouble decoding words. And because of the film, the video, 
that really drew them in, now that's in their mental Rolodex, and they can use that as a 
strategy. So instead of glossing over a word, or just pronouncing a word just because it 
has some of the same letters, and not really looking at it, is giving them an opportunity to 
slow down and take their time and try to decode it. 

The phonics skills were nice. I think they’ll remember the phonics skills, i think that was a 
good preparation. 

...on the DVD they talked a lot about like voice, like when they’re reading a story, they 
would act, like reading like a sad voice or a happy voice, and they’d be like, just like the 
guy in the video! So i think they picked up a lot more than i even knew they picked up 
during the sessions. 

But it definitely helped them with like the reading, like to slow down their reading and 
think about, like it talks a lot about some reading too quickly because you skip words. But 
they definitely focus on that a lot. The kids would. They were like i have to read slowly so 
I don’t miss a word like in the DVD. So i think so. i think it will better prepare them than 
when they were going in, definitely. 


Association Between Prankster Planet Usage and Student Learning Outcomes 

We investigated the association between Prankster Planet usage and growth on the standards/ 
skills aligned survey, and growth on the mathematics vocabulary section of the student inter¬ 
view. Prankster Planet usage was measured in hours and ranged from 0.09 hours to 7.79 hours. 
Google Analytics reported that the average student used Prankster Planet for 2.22 hours 
(SD = 1.60). As mentioned earlier, it is assumed that these data are under-reported for various 
reasons (see page 22). Growth on the standards/skills aligned survey, and on the mathematics 
vocabulary section of the student interview was calculated by subtracting the students’ scores 
at the post-test from their scores at the pre-test. We utilized multiple regression models to 
examine the association between Prankster Planet usage and growth on the two measures. In 
the multiple regression models, the difference scores on the two measures were the dependent 
variables. Prankster Planet usage in hours was entered as a predictor of the difference scores in 
both models. To statistically control for the students’ initial performance levels, we also entered 
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the pre-test scores from the corresponding surveys in the models. The multiple regression 
models revealed that Prankster Planet usage was not significantly related to growth on the stan¬ 
dards/skills aligned survey, or to growth on the mathematics vocabulary section of the student 
interview (both ps > .46). 

The lack of statistically significant associations between Prankster Planet usage and growth on 
either measure may be due to the relatively few hours that Google Analytics reported students 
played the game. In addition, Prankster Planet was but one aspect of the TEC curriculum. It is 
possible that other elements of TEC model, such as the number of episodes watched by stu¬ 
dents, would be associated with higher survey scores. However, associations between other 
elements of TEC and student outcomes cannot be explored because Prankster Planet was the 
only aspect of TEC model for which exposure was measured at the student level. 


DO STUDENTS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM INCREASE THEIR MOTIVATION AND CONFIDENCE TO 
ENGAGE IN LEARNING ACTIVITIES RELATED TO LITERACY? 

Ten out of 16 teachers reported that their students grew significantly in the area of “motiva¬ 
tion and confidence to engage in learning activities related to literacy.” Another four teachers 
reported that their students did progress in this area. Specific comments include: 

Oh, the word wall is great, because like I said, they enjoy—especially when they have the 
long words like "incredible’’ and "fabulous," all of those, they really get into it. And the 
word wall, they're using it when we do journals. They use the words in their journals, and 
when we do our activity of filling in the words, you know, reading the story and filling in 
the words. 

/ think it helped in making vocabulary more interesting and exciting to them. / had more 
students interested in learning what the unknown word was rather than skipping over it 
when reading. 

/ found many of them using the songs and lessons from the episodes to explain to their 
peers why certain words or sounds worked the way they did. 

It is a lot more interesting, because they didn’t realize they were learning in all those dif¬ 
ferent parts. So that’s what helped them a lot of times, and / think it also helped solidify 
the words with the motions. They also followed up with the show. We watched the same 
show the next day. Then they can focus more on the words and the different things that 
are going on once they’ve got the storyline down from the previous day. / really liked the 
process. / really liked the connection between the mental stimulation and the physical. 
We’re moving around, we're doing things. That helped them a lot because they are very 
active. 


( 


WestEd // Evaluation of The Electric Company Summer Learning Program // FINDINGS 


page 


41 





We could not assess whether the students’ attitudes about reading improved over the course 
of the TEC study because the subscale did not have adequate reliability. Future studies that 
investigate the TEC Summer Learning Program could modify the subscale or identify a different 
measure with acceptable reliability for the population under study. Due to the low reliability of 
the subscale, we opted to present the students’ responses only at post-test (see Exhibit 15). The 
students generally reported having positive attitudes about reading. For instance, 71.4% (n = 90) 
of the students indicated that they agreed that, "There are many books that I hope to read.” 

In addition, nearly two-thirds of the students reported that they disagreed with the statement, 
“Books are boring.” 


Exhibit 15: Students' Attitudes about Reading at Post-Test 



1 Agree 

1 Don’t Know 

1 Disagree 

Total 

Item 

% 

n 

% 

n 

% 

n 

% 

n 

1. There are many books that 1 
hope to read. 

71.4% 

90 

15.9% 

20 

12.7% 

16 

100.0% 

126 

2. There should be more time 
for free reading during the 
school day. 

52.8% 

67 

19.7% 

25 

27.6% 

35 

100.0% 

127 

3. Books make good presents. 

55.1% 

70 

18.1% 

23 

26.8% 

34 

100.0% 

127 

4. Books are boring. 

22.2% 

28 

11.9% 

15 

65.9% 

83 

100.0% 

126 

5. A certain amount of 
vacation time should be spent 
reading. 

48.0% 

60 

20.8% 

26 

31.2% 

39 

100.0% 

125 


DO STUDENTS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM INCREASE THEIR MOTIVATION AND CONFIDENCE TO 
ENGAGE IN LEARNING ACTIVITIES RELATED TO MATHEMATICS? 

Seven out of 16 teachers reported that their students grew significantly in the area of “motiva¬ 
tion and confidence to engage in learning activities related to mathematics,” while another 7 
reported that their students did progress in this area. Specific comments include: 

They went crazy measuring. They wanted to measure every single thing in the room. 
They were measuring heads and feet and hands. Well, my hand is—let me see if my hand 
is larger than yours, or smaller. You know? We talked about how you can use the small, 
smaller, smallest. / think they just love everything. 

Yesterday what we did was we were rolling the dice on the bulls eye and they were 
adding zeroes, either two zeroes or one zero. And initially I thought, oh, this is not going 
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to work well. And the kids were really excited and they kept comparing which numbers 
were bigger, which ones were smaller. They were just pumped. They were excited and 
they were ready to learn. 

They're motivated. They like the videos and they know that they mean the concepts, the 
math concepts are fine and they know that it means something. Yeah, and when they go 
home, they talk about it with their parents. They talk about the video that day. And what 
the words mean. 

We did not evaluate whether the students’ attitudes about mathematics improved over the 
course of the TEC study because the subscale did not have acceptable reliability. Consistent with 
our recommendations based on the reading attitudes subscale, future studies that investigate 
the TEC Summer Learning Program could modify the mathematics attitudes subscale or employ 
a different measure that has adequate reliability for the population being studied. Exhibit 16 
contains the students' responses at post-test to items on the mathematics subscale. The stu¬ 
dents reported fairly positive attitudes about mathematics. For instance, 72.6% (n = 90) of the 
students indicated that they agreed with the statement, “Math is interesting.” On the other hand, 
approximately equal numbers of students reported that they agreed and disagreed with the 
statement, “It is easy to get tired of math.” 

Exhibit 16: Students'Attitudes about Mathematics at Post-Test 



1 Agree 

1 Don’t Know 

1 Disagree 

Total 

Item 

% 

n 

% 

n 

% 

n 

% 

n 

1. Working math problems is 
fun, like solving a puzzle. 

69.8% 

88 

12.7% 

16 

17.5% 

22 

100.0% 

126 

2. It is easy to get tired of math. 

40.5% 

51 

14.3% 

18 

45.2% 

57 

100.0% 

126 

3. Working math problems is a 
waste of time. 

33.3% 

41 

12.2% 

15 

54.5% 

67 

100.0% 

123 

4. It is easy to understand 
math. 

59.2% 

74 

13.6% 

17 

27.2% 

34 

100.0% 

125 

5. Math is interesting. 

72.6% 

90 

17.7% 

22 

9.7% 

12 

100.0% 

124 


DO TEACHERS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM REPORT AN INCREASE IN MORALE AND ENTHUSIASM IN 
THEIR SUMMER SCHOOL SETTING? 

All 16 teachers participating in the TEC Summer Learning Program reported that using the pro¬ 
gram increased student morale and enthusiasm in their summer school setting. Many teachers 
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were both surprised and impressed with the level of engagement the program sparked in their 
students. Specific comments included: 

Students felt privileged to be part of the program. 

We have like four kids that I had worked with during the year who did not want to come 
to school, were just anti-school. And by the end, they were asking, can we have Electric 
Company today, we want to watch it! 

/ think whenever you say game, they’re like woooo! Game! Anything that we have, where 
we can keep score. They just love it. They really seem to be liking it. And they get excited 
after lunch, they line up, and they come up here. You can tell they’re getting excited. And 
they tell other kids what they’re doing. Like, we're going to The Electric Company today! 

/ don’t know if you heard them today, but / told them, today’s our last episode, and they 
were, what! The last episode! We don’t get to do it anymore? And they told me...tell the 
lady to send you more! Tell the lady to make more Electric Company/ 

A lot of the things that the people were doing in the show they really liked, because 
it wasn’t, you know, so kiddy, it was more like something that was cool...And even the 
songs, the monsters, it had a beat to it, so they were like okay, you know, it wasn’t some¬ 
thing say like Twinkle Little Star. They really liked that. 

The kids are engaged. They enjoy it. The music and just the way it’s presented makes you 
want to get involved. And / think for the kids, they really enjoyed going to the computers, 
playing the game. 

/ think they're just as excited this week as they were the first week. They haven’t gotten 
bored at all. And you see how attentive they are. 

/ mean / still had kids powering up...They were still powering up to the last episode. 
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TEACHER OUTCOMES 

This section addresses the research questions that concern teacher outcomes. Data from 
teacher interviews and exit surveys were used for these analyses. 


DO TEACHERS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM INCREASE THEIR MOTIVATION AND CONFIDENCE 
LEADING GROUPS? 

Fifteen out of 16 teachers reported that the TEC summer activities increased their motivation 
and confidence in leading groups. Specific comments included: 

The curriculum was very helpful in making direct instruction user friendly. 

Music and video always increase motivation for me. It was a lot of fun. 

/ think they—when they have an opportunity to do the think, hear, share, that they actually 
do support one another and have the conversations at the table or if we’re on the rug. 
They are respectful to one another to listen to the opinions. 

i think it's kind of hard for them to work together sometimes. So the group activities have 
been good reinforcers, so it’s good to have them work together with certain things. 

I liked the program and how the structure of it built throughout the summer. 


DO TEACHERS WHO PARTICIPATE IN THE TEC SUMMER LEARNING 
PROGRAM LEARN NEW INSTRUCTIONAL STRATEGIES IN TEACHING 
MATHEMATICS AND LITERACY? 

Nine out of 16 teachers reported that they learned new instructional strategies in teaching math¬ 
ematics and literacy. The three most common areas of learning were: new strategies for teaching 
vocabulary, incorporating technology into lessons, and combining different modes of learning to 
keep students engaged on the same topic. 

The overwhelming number of comments were about learning new instructional strategies 
related to implementing vocabulary instruction. Specifically, the use of Jessica’s Word Wall and 
word balls, and using movements for words were described by teachers as new and excellent 
reinforcements for acquiring new vocabulary. Comments included: 

/ loved how the vocabulary is introduced and remembered through Jessica’s Word Wall. 

...like kid friendly definitions. Some of those I’m going to bring back to the class, because 
they were pretty easy for the kids to understand and act out. / like that they put on their 
own movements for the words. 
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/ had heard about movement and vocabulary, but / had never used it in the way that we 
used it this time along, and i felt that the students really captured it. That’s something 
that I know I’m using next year. 

We’ve been growing, like the word ball thing. That’s something that we just found and 
thought was totally cool. 

Several teachers remarked that not only did their computer skills grow over the course of the 
summer, but that they became more comfortable with the idea of bringing technology into their 
classroom. Specific comments included: 

/ learned how to use technology to make the learning of math concepts fun and engaging 
for students. They taught me how to use the tools on Prankster Planet with greater ease. 

You know, we live in a society where they’re text messaging all the time. So being able to 
go to the computer and do some activities and then come back and apply it with pencil 
and paper is one way to help them see that there are times when you want to be learning 
things using pencil and paper, and there are times when you’re going to be using your 
hands punching keys... / already know how / want to use this in my workshop activity in 
the fall. 

/ mean, / had no idea about the videos, the video games that are available online for free. 
That was definitely new for me, that PBS web site. I’m mesmerized by it now. 

A few teachers reported that they appreciated how the TEC curriculum used all different kinds 
of groupings and activities to keep students engaged throughout. It made them think about how 
to structure their own classrooms during the regular school year. Specific comments included: 

There are a lot of activities in the curriculum that I’m definitely going to use again... 
Because the kids really need movement and activity and song and that kind of stuff. 

/ liked the way the routine went and the different components together. / felt like it keeps 
the kids going and moving and doing different things, but on the same topic. And / could 
see us using like some science and social studies as well. 


TEACHER FEEDBACK 


This section discusses findings regarding which aspects of the TEC model that summer school 
teachers felt both they and their students found the least and most appealing. Data from the 
teacher exit surveys, teacher interviews, and student interviews were used in the analyses. In 
addition, we present findings concerning whether teachers would use the TEC curriculum again 
(both in summer school and during the school year), and whether they would recommend the 
curriculum to other teachers. 
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WHICH ASPECTS OF THE TEC MODEL DO STUDENTS FIND THE 
MOST (AND LEAST) APPEALING, INTERESTING, AND EDUCATIONAL? 

Teachers were asked to rank the following six aspects of TEC curriculum on their appeal to 
students, level of student engagement, and educational value on a scale of 1 (least) to 6 (most): 
1) Session 7 of the episodes (watching it without interruption); 2) Session 2 of the episodes 
(pause and play session); 3) Prankster Planet: 4) Jessica's Word Wall; 4) teacher led activities; 
and 5) the activity worksheets. The results of the analysis appear in Exhibit 17. Teachers reported 
that students found Session 7 of the episodes to be the most appealing and engaging aspect 
of the model (mean = 4.94; SD= 1.24) and the activity worksheets to be the least engaging and 
appealing (mean = 1.63; SD = 1.15). Session 2 of the episodes was ranked by teachers as the 
second least appealing and engaging aspect of the model for students (mean = 2.94; SD= 1.24). 

In the analysis of the student interviews, we found that the double viewing of TEC episodes 
was both liked and disliked by students. For example, a group of students noted that the Pause 
and Play viewing with support activities (questions, games, and small group discussions) on the 
second viewing was beneficial to learning and retaining the core concepts: 


Interviewer: Which part of the episodes do you like better, do you like watching it the 
first time straight through, or watching it for the second time, after you’ve 
already seen it? 

Student: Watching it for the second time...Because sometimes I forget what 

happened. 


Interviewer: When you watch the episodes, do you like seeing it the first time or the 
second time better? 

Student: Both...[s]o I can remember about it and keep it in my head. 


However, other students disliked the interruptions of the Pause and Play viewing. The following 
student comments highlight this sentiment: 


Interviewer: What time do you like better, the first time or the second time? 
Student: The first one. 


Interviewer: Do you like those times. 

Student: No, our teacher like[s] stops sometimes, but I don’t want to stop. 

Interviewer: So do you like just seeing it the very first time all the way straight through, 
or do you like seeing it again for the second time. 

Student: / just want to see it all through. Not the second time. 

Finally, analysis of the student interviews revealed that four TEC episodes were particular favor¬ 
ites— “Pies for Puppies", "Spooky Summer Soiree", "Tip It or Dip It", and "Prankster Holiday." 
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Exhibit 17: Average Teacher Ranking of Elements of TEC Model on Appeal, Interest, 
and Educational Value to Students (n = 16) 


Aspect of the Model 

Mean Teacher 
Rank (1- 6) 

Standard 

Deviation 

Range 

0-6) 

Session 7 of the DVD Episodes 

4.94 

1.24 

2; 6 

Session 2 of the DVD Episodes 

2.94 

1.24 

l; 5 

Prankster Planet 

4.88 

1.82 

1; 6 

Jessica’s Word Wall 

3.44 

1.36 

1; 6 

Teacher-led Activities 

3.19 

0.83 

2; 4 

Activity Worksheets 

1.63 

1.15 

1; 5 


WHICH ASPECTS OF THE TEC MODEL DO TEACHERS FIND THE 
MOST (AND LEAST) APPEALING, INTERESTING, AND VALUABLE, 
PARTICULARLY IN REGARD TO LEARNING CONTENT AND INCREASING 
MOTIVATION? 

Teachers were asked to rank the following six aspects of TEC curriculum on their appeal, 
interest, and value on a scale of 1 (least) to 6 (most): 1) Session 7 of the DVD episodes (watching 
it without interruption); 2) Session 2 of the episodes (pause and play session); 3) Prankster 
Planet ; 4) Jessica’s Word Wall; 5) teacher led activities; and 6) the activity worksheets. The 
results of the analysis appear in Exhibit 18 and mirror the results of their rankings of the same 
elements when teachers were asked to take their students’ perspective. For example, teachers 
found Session 7 of the episodes to be the most appealing, interesting, and valuable aspect of the 
model (mean = 4.69; SD= 1.78) and the activity worksheets to be the least appealing, interesting, 
and valuable (mean = 2.38; SD = 1.45). Teachers reported that Session 2 of the episodes was the 
second least appealing, interesting, and valuable aspect of the model (mean = 2.63; SD= 1.78). 
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Exhibit 18: Average Teacher Ranking of Elements of TEC Model on Appeal, Interest, 
and Value (n = 16) 


Aspect of the Model 

Mean Teacher 
Rank (1- 6) 

Standard 

Deviation 

Range 

0-6) 

Session 7 of the DVD Episodes 

4.69 

1.78 

i; 6 

Session 2 of the DVD Episodes 

2.63 

1.78 

i; 6 

Prankster Planet 

3.19 

1.64 

i; 6 

Jessica’s Word Wall 

4.00 

1.32 

2; 6 

Teacher-led Activities 

4.13 

1.15 

2; 6 

Activity Worksheets 

2.38 

1.45 

1; 5 


TEC Episodes Session 1 and Session 2 

Teachers provided positive feedback on the appeal and value of the TEC episodes. Specific com¬ 
ments included: 

/ know the students are interested in the show, and I see all the hidden learning in the 
show and I like it myself. So I’m glad we had the opportunity to use it in our school. 

The DVDs are amazing. I think they are really strong because they’re very different from 
the original Electric Company [...] The structure really worked for them. 

Teachers mentioned the singing and dancing embedded in the show allow the students to con¬ 
nect with the content. In addition, music in the show was also exciting for the teachers. Teachers 
found the TEC content on the show to be appropriate for the first grade. Specific comments 
included: 

[TEC] motivated me to teach! I loved the music. There aren’t too many curricuiums that 
have music that goes with it, and it gets so exciting [...] I’ve had to search for music to go 
with all my units. This is all in one place. It’s fantastic. 

I mean a word ball flying through the air. That’s first grade! Somebody really had a first 
grader at home and knew what their kid was into. 

When asked about their preference for Session I and Session 2 of the episodes, Exhibit 18 shows 
that teachers valued Session 7 more than Session 2. Several teachers stated that they didn’t think 
it was necessary to watch the episode a second time, though they also noted that the children 
certainly enjoyed doing so. Specific comments included: 
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/ think there’s a better use of time such as focusing on specific clips, which would make 
more sense to me than watching the whole half hour all over again. [...] if you’re looking at 
this to be for students who really need that extra, that are deficient. They do tend to be 
more kinesthetic sort. Letting them sit in front of the TV is not necessarily going to help 
them remember more. Give them something to do. Change the modality. 

They prefer just to sit—they enjoy it. They really do. i can see the benefits of both stop¬ 
ping it because it’s fresh in their minds and they get realistic in the moment, but they still 
remember it if you just wait until the end. 

Some of the students don't like to have interruptions, or they want me to pause it at a 
place that we’re not supposed to pause it. [i ask the students] why do you want to pause 
there? Because i like to see this character, or i like the way they make the song come 
alive. Those were some of the comments that I heard from the kids. 

Also, one teacher mentioned when he watched the Session 7 episodes with his students he 
would note math and literacy activities he could add for the following day. Some teachers 
valued the classroom activities during the pause and play of the Session 2 episodes. One teacher 
commented: 

/ think pause and play is really powerful because it gives you the chance to kind of check 
in on those little segments. 

Some teachers reported that the episodes covered too many mathematical concepts for a 
summer program. Some students have not been exposed to the mathematics concept covered 
in the episodes, so watching the episodes was an introduction to that concept instead of a 
review. Teachers commented the students needed more time with difficult mathematics con¬ 
cepts, such as money. Teachers requested the curriculum cover fewer mathematical concepts in 
more depth. 


Teacher-Led Activities 

In general, teachers were impressed with the overall layout of the curriculum. A number of 
teachers commented how the curriculum guide was easy to use. Specific comments included: 

Everything is laid out really nicely for me. It’s user-friendly as far as the lessons and 
what's scripted, what I'm supposed to say or the materials / need. All that is nicely put 
together, so / haven’t had any problems with that. 

For me, the curriculum was aligned very nicely in the binder. It told me the materials / 
needed, my procedures, what / needed to say, what / needed to prepare. All of that stuff 
was very handy and organized and / could look at it the night before, the week before, 
and knew exactly what / was doing [.JThe routines are there so that it’s consistent. 
Whether you’re doing session one or session two, it’s the same thing every time. 
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According to Exhibit 18, teachers found the teacher-led activities to be the second most educa¬ 
tionally valuable component of the curriculum. Teachers noted that student motivation increased 
when they were able to connect what they watched in the episodes to the classroom activities: 

Just having that kind of whole-group connection to like when Marcus was doing such- 
and-such, or when the Pranksters did that [...] So apart from viewing it, and then learning 
it together, on the board or in the small group. They were excited when they were able to 
connect what they just saw to the classroom concepts. 

A number of teachers noted the value of the motions and vocabulary activity. One teacher com¬ 
mented about the teaching vocabulary with actions: 

Sometimes you can see which ones know it, which ones don’t really know it. Like they can 
all kind of give you a general definition. But then sometimes when it actually comes to 
acting out the words, you can really see who understands. 

One teacher mentioned that some classroom activities were not as educationally valuable as 
others. For example, she commented about the dot game: 

It was strictly a game. We weren’t grasping, where does the education come in here, 
where is the reinforcing what we’ve done. It was reinforcing what we've done to the 
terminology in the directions, but other than that it was strictly viewed as a game and / 
feel that was lost. 

In addition, a number of teachers modified the classroom activities for the Revenge of the Zeros 
episode because they felt the content was not mathematically sound. Teachers commented that 
the ‘adding zeros’ activity did not properly reinforce the concept of place value. One teacher 
modified the classroom activity by using place value cards instead of the prescribed TEC 
activity. Despite the variation in academic content in the classroom activities, teachers found the 
activities to be an important component to learning the TEC mathematics concepts. 


Jessica s Word Wall 

According to Exhibit 18, teachers reported Jessica’s Word Wall to be the third most valuable 
component of the TEC curriculum. A number of teachers commented on the difficulty level of 
the vocabulary words: 

/ like that the words that they're learning are not typical sight words, as we see them. / 
like the adding of collaboration, minute, compete. Bigger words that you’re hearing, so 
why not start you with those. [...] Because / think when they're given those words, they’re 
going to rise to that level. You know, water seeks it own level. They’re going to start 
functioning on that level. 

[The students] like learning. I tell them we are not using little baby words anymore. We 
are going to use big words. 


C 


WestEd // Evaluation of The Electric Company Summer Learning Program // FINDINGS 


page 


51 





Teachers mentioned that displaying vocabulary words is a common classroom strategy they use 
during their regular school year. The Word Ball cut outs were too small for students to see on 
the board. Researchers observed teachers writing the words on the board in larger print to allow 
all the students to see the electric words. 

Teachers found the review of the past vocabulary words with the Word Wall to be a valuable 
component to the curriculum. Specific comments included: 

[The Word Wall] worked incredibly. As long as we kept bringing back the words that they 
had already used, they could remember it. / wish they could do that in the show. 

/ would say the vocabulary words and the word wall. / definitely think that was really 
good. Because they are words that they are going to use throughout the year that they 
now have a better understanding going in. 

So the word wall activities are great because it's still reinforcing the words that have 
already been learned. They’re still having experience with those words. 

[The word wall] helped them think about the word, what did / see in that word ball. How 
was it used? They would remember the symbols. It wasn’t just here’s the definition, what’s 
the word? It was a lot of other components to it to help them really focus on it. So / think 
that helped. 

[I lovedj looking at how many times they used the vocabulary words in real life situations 
and having the kids have the opportunity to act it out, or define the word for us through 
mime or whatever was real fun. 


Prankster Planet 

According to Exhibit 18, teachers did not find the Prankster Planet game as valuable as the other 
components of the TEC curriculum. Most teachers enjoyed the game but wished more academic 
content was embedded in the game. In addition, teachers noticed that students were able to 
complete more missions if they were stronger readers and had more experience with computers 
in their homes. One teacher observed: 

If they could read what the characters were saying on Prankster Planet, / would have felt 
that they walked away with something more. But you know, we had to read a lot of it to 
them, and like when they got to the definitions part of it, they were able to tell me the 
definition of the word, but / had to read the choices to them. 

Teachers mentioned they wished the academic content came earlier on in the game to give 
students more opportunity for learning: 

It would be nice to see, either they have to do the words in order to get into the game, 
like an entry point, so the learning is frontloaded rather than back loaded [...J Instead of 
the having the five words at the end, you have to find this word here, and then you have 
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it throughout the game or frontioaded. So that way they’re not doing playing, playing, 
playing. Then, oh, sorry guys, its time to go, and they don’t ever get to that actual 
learning part. 

Teachers mentioned that students also had trouble navigating their character through the game. 
One teacher commented: 

So it was kind of hard for them to do with their motor skills, when you have to press the 
arrow and the spacebar. That was hard for them. Some of the students didn’t finish [a 
level] because they couldn't get to it. [...] If they couldn’t do it after a couple days, they 
would just move on to another mission. 

Students were able to select any mission on Prankster Planet and researchers often noticed 
students did not select the mission aligned with the day's episode. Teachers found students 
who would skip answering the questions at the end of the level to move on to another mission. 
Some teachers would make sure the students answered the questions at the end of the mis¬ 
sion, whereas some teachers had too many students to monitor the progress of their students 
through the game. 

Teachers enjoyed the overall appeal and variety of the game. Teachers noticed increased stu¬ 
dent engagement with the Prankster Planet game. Specific comments included: 

There were enough activities on Prankster that they could do games. You could do the 
words. / mean there were enough that you’re always doing something different. 

/ see Prankster Planet as reinforcement, when they’re really actively engaged. They’re 
being challenged, but there’s enough help that they’re able to find success. 


Activity Worksheets 

According to Exhibit 18, teachers valued the activity worksheets the least. Teachers noted the 
Session 1 worksheet was too short and students were able to complete the worksheet in less 
than 10 minutes. Students often did not read the passage and would guess which word was 
correct by using the letters given to them. Because the Session I worksheet was very short, a 
number of teachers included additional mathematics and literacy activities to supplement the 
TEC curriculum. A number of teachers mentioned students struggled reading the passages. 
Specific comments included: 

/ think it's just too many words. For a student who can't read—they see it and it's in 
a jumble. 

They have a hard time because the worksheets are all reading [...] They could tell me if 
/ asked and if / read it to them. They could pick it up that way, but they couldn't read 
it themselves. 
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Aside from the passages, teachers requested a greater variety of literacy and phonics exercises 
on the Session 1 worksheet to help support lower level readers. 

A number of teachers preferred the Session 2 worksheet to the Session 7 worksheet because 
the majority of the mathematics content was embedded in the Session 2 worksheet. Since most 
students were introduced to the mathematical content for the first time on TEC, the Session 2 
worksheet needed a substantial amount of teacher scaffolding in order to be completed by the 
student. One teacher mentioned the worksheets being the most difficult aspect of the cur¬ 
riculum to implement: 

/ think the worksheets are challenging for the students. They were challenging. Most of 
the time / had to sit with them. The form was meant for a small class, not for a class of 
22, and / was very fortunate to have student helpers, like high school and middle school 
helpers to take charge of the computers, and they were helping the students when / just 
did the worksheets. / think that worked well. 

In addition to the TEC small group activity, researchers observed teachers working with students 
individually to help struggling students with the worksheet. Some teachers repeated difficult 
worksheets as a class because all the students did not understand the mathematics concept. 
Teachers requested that the curriculum provide additional mathematics activities to help sup¬ 
port students who needed more practice learning the mathematics concept. 


WOULD TEACHERS USE THE SUMMER CURRICULUM AGAIN? WOULD 
THEY USE ALL OR PART OF THE CURRICULUM IN THEIR REGULAR 
CLASSROOM? WHY OR WHY NOT? 

Fifteen out of the 16 teachers reported that they would use TEC curriculum again, and 15 out of 
the 16 teachers reported that they would recommend the curriculum to other summer school or 
afterschool teachers. Comments include: 

It's a great program... If I would do it next year, / would definitely use this again. 

/ would recommend it, but not for like during the school year. / would recommend it for 
summer. I’d say yeah, this is a great program for the summer. In fact it has a lot of ideas. 
Like if you want to do any kind of program, / would definitely structure something around 
The Electric Company theme. 

It's been great for us to have this summer, and my co-workers that / work with during 
the summer program have asked if / get to keep the DVDs, because they wanted to keep 
them for teacher use, like this would be great for us to use again next summer. The kids 
are very excited about it...And so definitely with them being excited about it, / know that 
other teachers will be excited and willing to do the same program. 
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Thirteen out of 16 teachers reported that they would use all or part of the curriculum as part of 
their regular classroom activities. All three of the teachers who said they would not use it as part 
of regular classroom teaching stated that, while they would not use it as their primary teaching 
tool, they would use portions of the TEC curriculum to reinforce concepts already taught in the 
classroom. Comments include: 

/ would say that, as a teacher, / would definitely use the episodes, the pieces of the epi¬ 
sodes and take that and run with it, and then teach more in depth. 

The computer game, there's not enough time in a regular classroom...And / guess if it was 
my regular classroom / might have them act out words anyway. That is something that I'd 
do in my classroom. It gets them more engaged and they really have to think about the 
meaning. / think one thing / guess that / would use, is, you know, drawing the pictures and 
then writing the caption next to it. 

/ don’t think our district would allow it... It just seems like it’s material to use in a review. 

/ think something that I’d want to use in my classroom now is the movements for 
vocabulary words. And to me, it was like oh wow, the movements really get to the kids 
and that’s something I’m planning to use next year in my own regular classroom. 


SUGGESTIONS FOR IMPROVEMENT 

Data from both the teacher exit surveys and interviews informed this section. We have divided 
the section into subcategories related to professional development and the TEC Summer 
Learning Program curriculum. 


WHICH ASPECTS, IF ANY OF THE TEC MODEL (INCLUDING 
PROFESSIONAL DEVELOPMENT) DO TEACHERS THINK MIGHT NEED 
TO BE REFINED, IMPROVED, OR MODIFIED? 

Professional Development 

When asked on the teacher exit survey about suggestions for refining, improving, or modifying 
the professional development, six teachers did not respond while another four expressed no 
need for curriculum modifications. The responses from the other six teachers varied greatly and 
offered the following suggestions: include video clips or allow teachers to see the curriculum 
being implemented in an actual classroom; include the list of vocabulary words in the begin¬ 
ning of the curriculum manual; dedicate the webinars to questions and answers only (i.e., have 
teachers review the materials ahead of time); and schedule webinars later in the day for teachers 
on the west coast. During a teacher interview, one teacher recommended an additional webinar, 
“Having a midsession webinar to see what the other people were doing and compare notes. This 
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is what we have a problem with. Oh we had that problem too and we fixed it by doing this. It 
becomes more of a collaboration.” 

Several teachers noted the repetitive nature of the professional development webinars with 
the curriculum guide. In order to break the repetition, a number of teachers suggested that the 
professional development include video clips to allow teachers to view the curriculum being 
implemented in an actual classroom. Specific comments included: 

Honestly, / didn't find [the professional development] very helpful. / felt like everything 
that they were telling us was stuff that was on the manual, and I felt like they were just 
reading it to me. / felt like / could read this myself. / think it would have been more helpful 
if we actually got to see a classroom and the lesson being taught. Because that way 
you get ideas as to what is working and then / can make this change that will help my 
classroom. 

/ think [the professional development] was somewhat helpful. But / think if we saw it in an 
actual class, I think that would have been better because then you can get an idea of what 
are things you are supposed to do. 

Session I [of the professional development] was kind of just repetitive. / just sat there and 
felt like / had already gotten a good sense of what to do by just looking at the manual. If / 
didn’t look at the manual, then session one would have been helpful. 


Curriculum 

Teachers were asked how the Sesame Workshop might improve the curriculum and if there were 
any aspects of the TEC model that should be refined or modified. The aspect of the TEC model 
that evoked the most comments was the worksheets, which corresponds with the aforemen¬ 
tioned finding that the worksheets were ranked as the least engaging, valuable, or beneficial by 
teachers. A number of teachers reported the worksheets were often too difficult for students 
because of the reading involved, and several teachers suggested more practice with the pho¬ 
nics skills covered in the episodes. Two teachers suggested that more variation be built into 
the worksheets or that they revolve around project-based work. Teachers with longer summer 
school days were able to supplement the curriculum with their own teacher-developed literacy 
activities. Specific comments from teacher interviews included: 

/ actually would like to see them, like the G-H, P-H today, I'd love to see them do more 
with that, because that’s where a lot of our kids—I love the worksheets, / love the word 
wall, but they’re here because they’re not reading. / don’t expect them to spell the words, 
/ expect them to be able to know what it means. But / think to emphasize more on the 
phonics piece, especially if it's the first and second grade level, like as an intervention 
you’re talking about, you need to focus a little bit more on the phonics. 
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/ mean you saw how the kids loved the F and G-H and P-H and it was beautiful. If there 
were either worksheets or games or something that would incorporate it, to emphasize 
that, for them to rethink it on their own without the TV in front of them, / think that would 
be really useful. 

Most of them are in here because their parents want them to improve their reading, so 
that's why / was thinking like more. They do have phonics in the video, but maybe more 
literacy sheets. 

/ would change that the sentences and fill in the words was the activity every time. / 
would try to do it a little different. Maybe a word search. Maybe in a crossword puzzle. 
What is two down? What means straight. [...] Maybe giving synonym or a homonym. 
Depending on the words. Like if the word was pale and pail, for example. Which word 
would fit in a sentence. So give like three of them. Or give us, like one of the words, make 
little words from this one as a fun activity. 

Sometimes the challenging pieces were probably for, in this grade, because of limited 
reading ability, or their reading levels. Sometimes the worksheets cannot be independent, 
sometimes that was challenging for the students. / would have to read the captions for 
them, so they would know what was being looked for. [...] I feel that is difficult to follow 
multiple directions, especially if it's meant to be independent. 

Several teachers reported the content covered in the curriculum was often an introduction to the 
mathematical concepts and not review for the students. According to the teacher survey, three 
teachers mentioned the worksheets were not appropriate for the ability levels of their students. 
One teacher commented: 

A lot of the games are adding numbers over a hundred and some of them just aren’t 
ready for that yet. They needed a lot of support. Some of the games they weren’t able to 
complete independently because it focused more on skills that are learned in 2nd grade 
versus reinforcing skills that they learned in first grade. 

Teachers often needed to supplement the curriculum with additional instruction and commented 
there was not enough time with each mathematical topic. Teachers reported the second work¬ 
sheet to be especially difficult for students because of the mathematical content. Due to the 
time constraints of the summer, several teachers requested to cover fewer mathematical con¬ 
cepts in more depth and to allow students additional time to practice the new mathematical skill. 
Teachers requested the curriculum offer more worksheets and activities with difficult concepts, 
such as money, place value, and time. Specific comments from teacher interviews included: 

If we’re going to use it as a summer school program, then we might focus on one or two 
concepts, because for this four-week session, they were telling time, measurements, 
graphing, and / think problem solving. [...] Because it’s four weeks, for some kids who 
really haven’t grasped the concept, if we focused on it for a little longer period of time. 
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[...] Just have those episodes continue to relate to telling time and really going into detail 
about 15 minutes, 45, 30. 

/ don't think graphing, because graphing is a thing that’s so easy. I don’t think graphing 
needs to be reviewed over the summer, because they just breezed through that. But 
definitely addition and subtraction within ten, place value to a hundred, time, money and 
measurement would be the main concepts to focus on. 

So when they were trying to add up the cents, they just couldn’t do it. [...] They were get¬ 
ting very frustrated. So we kind of modified it, and just focused on how much is a quarter, 
how much is a dime, rather than the whole activity. They struggle. 

Just the coins. Like yesterday, they were just sitting there going like, / don’t get it! And / 
even brought in coins to actually add it. But it was just too much at once / think. Because 
they are just learning adding. So when we threw in the coin piece, they were like, wait. 
We’re still learning like six plus one is seven. 

I had to do a lot of supplementing. But you know, / think if it was geared more to the 
lower levels, a lot more practice facts, facts within ten, and a lot more practice place 
value, / think that would be an improvement for the program. 

If we’re doing a particular math concept, that we would continue with, say, graphing or 
measuring or two-digit addition or something like that. And if we’re in a four-week or 
six-week session, use that particular concept, or maybe two. And that way we can really 
make certain that we know that the kids understand it from many different perspectives, 
in terms of graphing or measuring items, or if they’re dealing with money. Different ways 
to count it and write. Sometimes / felt like we were doing too many math concepts in a 
four-week period. 

According to the interviews, teachers reported the Word Wall ball cut outs were often too small 
for the students to read in the classroom and teachers had to modify balls to allow all students 
access to viewing the words. One teacher suggested a glowing ball be included in the curriculum 
to bring the content from the show to the classroom and improve student engagement with the 
vocabulary activities. Specific teachers comments included: 

Making the word wall was okay. The balls are small. You couldn’t really see them. If I used 
my own index cards and put them on my own, it would have been better, because they 
would have been bigger. They really couldn’t see them from the whole place. We had the 
wall, and they could go up to them, but it wasn’t till I pointed to [the words] did we say 
them. 

In addition, teachers reported vocabulary strategies should also be incorporated into the cur¬ 
riculum. One teacher commented: 
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To add that sort of layer in there, so that ELL’s recognize that this is why it’s helping me, 
because then there's that built-in strategy that maybe they can internalize and learning 
and switch around, so it’s something else that they can use outside of The Electric 
Company. Which, let’s face it, is really what we're trying to do. Five words in an hour and 
a half is not a whole lot of learning. We want them to learn the strategies of what they're 
doing, so that way when they come across those things, when they find them, they’re like, 

/ know how to figure that, because / learned how to do that with The Electric Company. 
Because that’s really where the learning happens when you are able to change it over into 
something real world. 

Teachers reported students had difficulty navigating through the Prankster Planet game because 
of low technological and reading abilities. Because of inexperience using computers, students 
would spend the entire computer time jumping around the game without ever reaching the 
academic content. Teachers requested more academic content be embedded at the beginning 
and middle sections of the game. Though the game was open for students to play any mission, 
teachers recommended the game be locked, so students are progressing level by level and 
reinforcing the skills from the day’s episode. Specific teachers comments included: 

When they go to Prankster Planet, if there was something in the teacher’s guide that 
said, in episode 12, these are the things that they will be working on, because some of the 
kids were in different places, and when / would try to get to them to see what they were 
doing, I'm like, well, is this what we did in class today? That was difficult for me to keep 
track of. 

The other thing / would say is, because quite often what they would do, is they would, say 
they would guess what the word meant, and they’d guess wrong, and then they’d have to, 
okay, it’s not this button. So it was more of this button doesn’t count anymore so it’s one 
of these two and / think that / might be more educationally valuable, instead of saying, it’s 
not the pink button, it’s going to be the blue or the green. Okay. Each time it has to re-ask 
the question or shuffle the answers so they’re starting to read the answers instead of just 
saying, okay, it’s not the pink, it's not the red button. 

Though participating teachers had a number of comments on how to modify the TEC curriculum, 
their overall experience with the curriculum was extremely positive. 
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The results suggest that the TEC Summer Learning Program can be a valuable tool for educa¬ 
tors to boost student interest in learning and to promote academic achievement in specific 
content areas related to numeracy and literacy. The evaluation focused on the use of the TEC 
Summer Learning Program in summer school programs serving students ages 6 to 8 years from 
low-income families. Students and teachers in the study found TEC to be engaging throughout 
the entire 24 days included in the model. The findings from this mixed-methods formative study 
suggest that the TEC Summer Learning Program can increase children’s interest in learning 
in specific subjects, and can contribute to increased academic achievement in numeracy and 
literacy. In addition to gains in student outcomes, teachers who used the curriculum reported 
gains in their own learning, particularly around ways to deliver instruction and technology use. 
Specific findings in the areas of student outcomes, teacher outcomes, program implementation 
and teacher feedback include: 

Student Outcomes: 

Students participating in the TEC Summer Learning Program showed significant growth in 
their knowledge of math vocabulary, numeracy skills, and literacy skills. 

Fourteen out of 16 participating teachers reported that students progressed or significantly 
grew in the area of “motivation and confidence to engage in learning activities” related to 
both mathematics and literacy. 

All 16 teachers participating in the TEC Summer Learning Program reported that using the 
program increased student morale and enthusiasm in their summer school setting. 

Teacher Outcomes: 

Fifteen out of 16 teachers reported that the TEC summer activities increased their motiva¬ 
tion and confidence in leading groups. 

Nine out of the 16 participating teachers reported that they learned new instructional 
strategies in teaching mathematics and literacy. The three most common areas of learning 
were: new strategies for teaching vocabulary, incorporating technology into lessons, and 
combining different modes of learning to keep students engaged in a topic. 

Program Implementation: 

Despite the diversity of the programs (with regard to class size, teacher experience, type 
of program, etc.), nearly all participating teachers implemented the TEC Summer Learning 
Program as planned. 
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Student outcomes were consistently positive, even with variations in the training teachers 
received and the delivery of the number of episodes of The Electric Company. 

Teacher Feedback: 

Fifteen out of the 16 teachers reported that they would use the TEC curriculum again, and 
15 out of the 16 teachers reported that they would recommend the curriculum to other 
summer school or afterschool teachers. 

Thirteen out of 16 teachers reported that they would use all, or part, of the curriculum as 
part of their regular classroom activities. All three of the teachers who said they would not 
use it as part of regular classroom teaching stated that, while they would not use it as their 
primary teaching tool, they would use portions of the TEC curriculum to reinforce concepts 
already taught in the classroom. 

This comprehensive mixed-methods study produced important findings about the TEC 
Summer Program and its effects on students, particularly in relation to its effects on stu¬ 
dents' classroom engagement, interest in academic content, and student learning in the 
summer school environment. The TEC curriculum is unique in that it is combines different 
modes of instruction (television show episodes, small and large group hands-on activities, 
and online gaming) to engage students in the academic content. It is intended to be deliv¬ 
ered in the Out of School Time (OST) environment, yet support students’ learning of core 
academic standards. This study is important in that is shows that an innovative transmedia 
summer program can fit seamlessly into the summer school environment and support 
schools as they strive to engage and teach students from widely diverse backgrounds and 
ability levels. 

The current study highlights the promise of transmedia in OST environments for student 
learning and success. The study also shows that teachers can successfully use and embrace 
a transmedia curriculum. The findings suggest that further study is warranted to explore 
how students learn in the TEC curricular environment. Topics for future study include: the 
added value of each aspect of the TEC Summer Learning Program (television show epi¬ 
sodes, small and large group hands-on activities, and online gaming), how teachers expand 
on the TEC curriculum, how teachers use aspects of the TEC curriculum in their regular 
classrooms, expansion of the TEC curriculum to include new digital devices, and use of the 
TEC curriculum in other OST environments, such as in after-school programs. 


RECOMMENDATIONS 

The following are recommendations based on the findings of this formative evaluation study: 
Implementation 

» Explore the use of the TEC curriculum in other OST settings. 
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» Continue research around how young children (particularly those from low-income 

families) learn in the TEC curriculum, including how transmedia enhances the curriculum. 

Professional Development 

» Explore different formats and structures of the orientation webinars. Suggestions 
include: a question and answer format after teachers have gone through curriculum 
materials themselves; viewing/studying an actual classroom implementing the 
TEC curriculum. 

» Explore the use of ongoing professional development for implementing the TEC cur¬ 
riculum. For example, having biweekly webinars or conference calls could help teachers 
share their ideas and/or concerns with regard to facilitation of the curriculum. 

Production 

» Explore the goals behind and content of the worksheet aspect of the TEC curriculum. 
The worksheets received the lowest ratings by the teachers who implemented the 
curriculum. 
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